2024-05-09T04:49:37.359 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-09T04:49:37.359 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi/7699557 --owner scheduled_vshankar@teuthology --name vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/backtrace} --verbose -- /home/teuthworker/archive/vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi/7699557/orig.config.yaml 2024-05-09T04:49:37.405 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-09T04:49:37.490 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi/7699557 branch: wip-63259-3 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/backtrace} email: vshankar@redhat.com first_in_suite: false job_id: '7699557' kernel: client: sha1: distro flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-63259-3 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy sha1: 165a15e51d08073b5317c443415b0d8a7141b1ab ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 165a15e51d08073b5317c443415b0d8a7141b1ab kclient: syntax: v2 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-63259-qa-fix-1 sha1: a2e1088fa282408acd6a8a29f9417b763c0a8663 owner: scheduled_vshankar@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 9329 sha1: 165a15e51d08073b5317c443415b0d8a7141b1ab sleep_before_teardown: 0 subset: 712/1024 suite: fs suite_branch: wip-63259-qa-fix-1 suite_path: /home/teuthworker/src/github.com_vshankar_ceph_a2e1088fa282408acd6a8a29f9417b763c0a8663/qa suite_relpath: qa suite_repo: https://github.com/vshankar/ceph.git suite_sha1: a2e1088fa282408acd6a8a29f9417b763c0a8663 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi060.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDXKlNmw7xjovvOPA5uoX35oogqy+fhjoG7qlpocOBTDo0OuCLVF4iqXfTNQm2BHg0xoBUU7UPRyEg01wz44wcU= smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIuN6mPI387qZJmyDmYmuGL/2LfVuKpBV7FADGpOtM1VFIMlHQuH6yKHxSB1OYXGM+A1VzzuYysAGwLIo0zDxDA= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_backtrace teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-09_04:42:47 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-09T04:49:37.491 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_vshankar_ceph_a2e1088fa282408acd6a8a29f9417b763c0a8663/qa; will attempt to use it 2024-05-09T04:49:37.492 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_vshankar_ceph_a2e1088fa282408acd6a8a29f9417b763c0a8663/qa/tasks 2024-05-09T04:49:37.492 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-09T04:49:37.493 INFO:teuthology.task.internal:Checking packages... 2024-05-09T04:49:37.516 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '165a15e51d08073b5317c443415b0d8a7141b1ab' 2024-05-09T04:49:37.516 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-09T04:49:37.517 INFO:teuthology.packaging:ref: None 2024-05-09T04:49:37.517 INFO:teuthology.packaging:tag: None 2024-05-09T04:49:37.517 INFO:teuthology.packaging:branch: wip-63259-3 2024-05-09T04:49:37.517 INFO:teuthology.packaging:sha1: 165a15e51d08073b5317c443415b0d8a7141b1ab 2024-05-09T04:49:37.517 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-63259-3 2024-05-09T04:49:37.743 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3568.g165a15e5 2024-05-09T04:49:37.744 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-09T04:49:37.751 INFO:teuthology.task.internal:no buildpackages task found 2024-05-09T04:49:37.751 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-09T04:49:37.778 INFO:teuthology.task.internal:Saving configuration 2024-05-09T04:49:37.794 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-09T04:49:37.811 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T04:49:37.837 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi/7699557', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 04:44:35.371617', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDXKlNmw7xjovvOPA5uoX35oogqy+fhjoG7qlpocOBTDo0OuCLVF4iqXfTNQm2BHg0xoBUU7UPRyEg01wz44wcU='} 2024-05-09T04:49:37.857 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi/7699557', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 04:44:35.370115', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIuN6mPI387qZJmyDmYmuGL/2LfVuKpBV7FADGpOtM1VFIMlHQuH6yKHxSB1OYXGM+A1VzzuYysAGwLIo0zDxDA='} 2024-05-09T04:49:37.857 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-09T04:49:37.864 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-05-09T04:49:37.864 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-05-09T04:49:37.864 INFO:teuthology.run_tasks:Running task console_log... 2024-05-09T04:49:37.961 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff17dee9f70>, signals=[15]) 2024-05-09T04:49:37.962 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-09T04:49:37.968 INFO:teuthology.task.internal:Opening connections... 2024-05-09T04:49:37.968 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2024-05-09T04:49:37.970 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T04:49:38.048 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T04:49:38.049 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T04:49:38.126 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-09T04:49:38.133 DEBUG:teuthology.orchestra.run.smithi060:> uname -m 2024-05-09T04:49:38.152 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2024-05-09T04:49:38.152 DEBUG:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2024-05-09T04:49:38.210 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="CentOS Stream" 2024-05-09T04:49:38.210 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="9" 2024-05-09T04:49:38.210 INFO:teuthology.orchestra.run.smithi060.stdout:ID="centos" 2024-05-09T04:49:38.210 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE="rhel fedora" 2024-05-09T04:49:38.210 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="9" 2024-05-09T04:49:38.210 INFO:teuthology.orchestra.run.smithi060.stdout:PLATFORM_ID="platform:el9" 2024-05-09T04:49:38.210 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T04:49:38.210 INFO:teuthology.orchestra.run.smithi060.stdout:ANSI_COLOR="0;31" 2024-05-09T04:49:38.210 INFO:teuthology.orchestra.run.smithi060.stdout:LOGO="fedora-logo-icon" 2024-05-09T04:49:38.210 INFO:teuthology.orchestra.run.smithi060.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T04:49:38.211 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://centos.org/" 2024-05-09T04:49:38.211 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T04:49:38.211 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T04:49:38.211 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T04:49:38.211 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2024-05-09T04:49:38.236 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-05-09T04:49:38.254 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-05-09T04:49:38.254 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-05-09T04:49:38.311 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-05-09T04:49:38.311 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="9" 2024-05-09T04:49:38.311 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-05-09T04:49:38.311 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-05-09T04:49:38.311 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="9" 2024-05-09T04:49:38.311 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el9" 2024-05-09T04:49:38.311 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T04:49:38.312 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-05-09T04:49:38.312 INFO:teuthology.orchestra.run.smithi114.stdout:LOGO="fedora-logo-icon" 2024-05-09T04:49:38.312 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T04:49:38.312 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-05-09T04:49:38.312 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T04:49:38.312 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T04:49:38.312 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T04:49:38.312 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-05-09T04:49:38.336 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-09T04:49:38.345 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-09T04:49:38.370 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-09T04:49:38.370 DEBUG:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2024-05-09T04:49:38.373 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-05-09T04:49:38.389 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-09T04:49:38.394 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-09T04:49:38.395 DEBUG:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2024-05-09T04:49:38.430 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-05-09T04:49:38.539 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-09T04:49:38.594 INFO:teuthology.run_tasks:Running task kernel... 2024-05-09T04:49:38.615 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-05-09T04:49:38.616 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2024-05-09T04:49:38.616 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2024-05-09T04:49:38.616 DEBUG:teuthology.orchestra.run.smithi060:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T04:49:38.617 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T04:49:38.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T04:49:38.635 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-05-09T04:49:38.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T04:49:38.637 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-05-09T04:49:38.693 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-437.el9.x86_64 2024-05-09T04:49:38.693 INFO:teuthology.task.kernel:Running kernel on smithi114: 5.14.0-437.el9.x86_64 2024-05-09T04:49:38.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum install -y kernel 2024-05-09T04:49:38.696 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-437.el9.x86_64 2024-05-09T04:49:38.696 INFO:teuthology.task.kernel:Running kernel on smithi060: 5.14.0-437.el9.x86_64 2024-05-09T04:49:38.696 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum install -y kernel 2024-05-09T04:49:41.269 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 8.9 MB/s | 8.1 MB 00:00 2024-05-09T04:49:41.964 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.1 MB 00:01 2024-05-09T04:49:45.573 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 7.5 MB/s | 19 MB 00:02 2024-05-09T04:49:49.824 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-05-09T04:49:52.133 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-05-09T04:49:54.043 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-05-09T04:49:58.751 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:16 2024-05-09T04:49:59.226 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-05-09T04:50:01.232 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-09T04:50:01.233 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-09T04:50:01.328 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-05-09T04:50:01.334 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-05-09T04:50:01.334 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-05-09T04:50:01.334 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-05-09T04:50:01.334 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-05-09T04:50:01.334 INFO:teuthology.orchestra.run.smithi114.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-09T04:50:01.334 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-05-09T04:50:01.334 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-09T04:50:01.335 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-09T04:50:01.335 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-09T04:50:01.335 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:50:01.335 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-05-09T04:50:01.335 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-05-09T04:50:01.335 INFO:teuthology.orchestra.run.smithi114.stdout:Install 4 Packages 2024-05-09T04:50:01.335 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:50:01.336 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 78 M 2024-05-09T04:50:01.336 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 123 M 2024-05-09T04:50:01.336 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-05-09T04:50:02.153 INFO:teuthology.orchestra.run.smithi114.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 718 kB/s | 484 kB 00:00 2024-05-09T04:50:04.469 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.2 MB 00:02 2024-05-09T04:50:06.111 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-05-09T04:50:07.879 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-05-09T04:50:13.188 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-05-09T04:50:15.319 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-09T04:50:15.319 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-09T04:50:15.416 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-05-09T04:50:15.425 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-05-09T04:50:15.425 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-05-09T04:50:15.425 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-05-09T04:50:15.425 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-05-09T04:50:15.425 INFO:teuthology.orchestra.run.smithi060.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-09T04:50:15.426 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-05-09T04:50:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-09T04:50:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-09T04:50:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-09T04:50:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:50:15.426 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-05-09T04:50:15.426 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-05-09T04:50:15.426 INFO:teuthology.orchestra.run.smithi060.stdout:Install 4 Packages 2024-05-09T04:50:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:50:15.426 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 78 M 2024-05-09T04:50:15.426 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 123 M 2024-05-09T04:50:15.426 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-05-09T04:50:15.714 INFO:teuthology.orchestra.run.smithi060.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 2.8 MB/s | 484 kB 00:00 2024-05-09T04:50:16.227 INFO:teuthology.orchestra.run.smithi114.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm 1.0 MB/s | 15 MB 00:14 2024-05-09T04:50:16.448 INFO:teuthology.orchestra.run.smithi060.stdout:(2/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 38 MB/s | 34 MB 00:00 2024-05-09T04:50:16.615 INFO:teuthology.orchestra.run.smithi060.stdout:(3/4): kernel-core-5.14.0-444.el9.x86_64.rpm 14 MB/s | 15 MB 00:01 2024-05-09T04:50:16.990 INFO:teuthology.orchestra.run.smithi060.stdout:(4/4): kernel-modules-core-5.14.0-444.el9.x86_6 22 MB/s | 28 MB 00:01 2024-05-09T04:50:16.991 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-05-09T04:50:16.991 INFO:teuthology.orchestra.run.smithi060.stdout:Total 50 MB/s | 78 MB 00:01 2024-05-09T04:50:17.705 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-05-09T04:50:18.097 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-05-09T04:50:18.097 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-05-09T04:50:19.042 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-05-09T04:50:19.042 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-05-09T04:50:21.629 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-05-09T04:50:22.280 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-09T04:50:22.358 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T04:50:25.184 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T04:50:25.314 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T04:50:30.393 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T04:50:30.433 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:50:33.317 INFO:teuthology.orchestra.run.smithi114.stdout:(3/4): kernel-modules-core-5.14.0-444.el9.x86_6 919 kB/s | 28 MB 00:31 2024-05-09T04:50:35.617 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:50:39.225 INFO:teuthology.orchestra.run.smithi114.stdout:(4/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 920 kB/s | 34 MB 00:37 2024-05-09T04:50:39.225 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-05-09T04:50:39.225 INFO:teuthology.orchestra.run.smithi114.stdout:Total 2.1 MB/s | 78 MB 00:37 2024-05-09T04:50:39.902 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-05-09T04:50:40.249 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-05-09T04:50:40.249 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-05-09T04:50:41.140 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-05-09T04:50:41.140 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-05-09T04:50:43.546 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-05-09T04:50:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-09T04:50:44.260 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T04:50:46.926 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T04:50:47.071 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T04:50:51.760 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T04:50:51.798 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:50:56.482 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:51:07.000 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:51:07.041 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:51:07.391 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:51:07.391 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-09T04:51:07.391 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T04:51:07.391 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T04:51:08.214 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:51:08.214 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:51:08.214 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-05-09T04:51:08.214 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-09T04:51:08.214 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-09T04:51:08.214 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-09T04:51:08.214 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-09T04:51:08.214 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:51:08.215 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-05-09T04:51:08.568 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T04:51:08.705 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T04:51:08.705 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T04:51:08.705 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-09T04:51:08.706 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-09T04:51:08.706 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-09T04:51:08.706 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-05-09T04:51:08.706 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi060.front.sepia.ceph.com, path=None, version=distro) 2024-05-09T04:51:08.706 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T04:51:08.759 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T04:51:08.759 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T04:51:08.759 DEBUG:teuthology.orchestra.run.smithi060:> sudo rpm -qi grub2-tools 2024-05-09T04:51:08.796 INFO:teuthology.orchestra.run.smithi060.stdout:Name : grub2-tools 2024-05-09T04:51:08.796 INFO:teuthology.orchestra.run.smithi060.stdout:Epoch : 1 2024-05-09T04:51:08.796 INFO:teuthology.orchestra.run.smithi060.stdout:Version : 2.06 2024-05-09T04:51:08.796 INFO:teuthology.orchestra.run.smithi060.stdout:Release : 68.el9 2024-05-09T04:51:08.796 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: x86_64 2024-05-09T04:51:08.797 INFO:teuthology.orchestra.run.smithi060.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-09T04:51:08.797 INFO:teuthology.orchestra.run.smithi060.stdout:Group : Unspecified 2024-05-09T04:51:08.797 INFO:teuthology.orchestra.run.smithi060.stdout:Size : 8264254 2024-05-09T04:51:08.797 INFO:teuthology.orchestra.run.smithi060.stdout:License : GPLv3+ 2024-05-09T04:51:08.797 INFO:teuthology.orchestra.run.smithi060.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-09T04:51:08.797 INFO:teuthology.orchestra.run.smithi060.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-09T04:51:08.797 INFO:teuthology.orchestra.run.smithi060.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-09T04:51:08.797 INFO:teuthology.orchestra.run.smithi060.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-09T04:51:08.797 INFO:teuthology.orchestra.run.smithi060.stdout:Packager : builder@centos.org 2024-05-09T04:51:08.797 INFO:teuthology.orchestra.run.smithi060.stdout:Vendor : CentOS 2024-05-09T04:51:08.797 INFO:teuthology.orchestra.run.smithi060.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-09T04:51:08.797 INFO:teuthology.orchestra.run.smithi060.stdout:Summary : Support tools for GRUB. 2024-05-09T04:51:08.797 INFO:teuthology.orchestra.run.smithi060.stdout:Description : 2024-05-09T04:51:08.797 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:51:08.797 INFO:teuthology.orchestra.run.smithi060.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-09T04:51:08.798 INFO:teuthology.orchestra.run.smithi060.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-09T04:51:08.798 INFO:teuthology.orchestra.run.smithi060.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-09T04:51:08.798 INFO:teuthology.orchestra.run.smithi060.stdout:hardware devices. 2024-05-09T04:51:08.798 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:51:08.798 INFO:teuthology.orchestra.run.smithi060.stdout:This subpackage provides tools for support of all platforms. 2024-05-09T04:51:08.799 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-09T04:51:08.799 INFO:teuthology.task.kernel:Updating grub on smithi060 to boot 5.14.0-444.el9.x86_64 2024-05-09T04:51:08.799 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-09T04:51:09.409 INFO:teuthology.orchestra.run.smithi060.stderr:Generating grub configuration file ... 2024-05-09T04:51:10.828 INFO:teuthology.orchestra.run.smithi060.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-09T04:51:10.856 INFO:teuthology.orchestra.run.smithi060.stderr:done 2024-05-09T04:51:10.859 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-05-09T04:51:10.879 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.r4SPJFYOdc 2024-05-09T04:51:10.879 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.r4SPJFYOdc 2024-05-09T04:51:10.954 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.r4SPJFYOdc 2024-05-09T04:51:11.099 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.r4SPJFYOdc is 6KB 2024-05-09T04:51:11.119 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.r4SPJFYOdc 2024-05-09T04:51:11.134 DEBUG:teuthology.orchestra.run.smithi060:> sudo /bin/ls /boot/loader/entries || true 2024-05-09T04:51:11.208 INFO:teuthology.orchestra.run.smithi060.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-09T04:51:11.208 INFO:teuthology.orchestra.run.smithi060.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-09T04:51:11.208 INFO:teuthology.orchestra.run.smithi060.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-09T04:51:11.208 INFO:teuthology.orchestra.run.smithi060.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-09T04:51:11.209 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-09T04:51:11.364 DEBUG:teuthology.orchestra.run.smithi060:> sudo shutdown -r now 2024-05-09T04:51:26.032 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:51:26.070 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:51:26.444 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:51:26.444 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-09T04:51:26.444 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T04:51:26.445 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T04:51:27.277 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:51:27.277 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:51:27.277 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-05-09T04:51:27.277 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-09T04:51:27.277 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-09T04:51:27.277 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-09T04:51:27.277 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-09T04:51:27.277 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:51:27.277 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-05-09T04:51:27.683 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T04:51:27.804 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T04:51:27.804 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T04:51:27.804 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-09T04:51:27.804 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-09T04:51:27.804 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-09T04:51:27.805 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2024-05-09T04:51:27.805 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=None, version=distro) 2024-05-09T04:51:27.805 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T04:51:27.855 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T04:51:27.855 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T04:51:27.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2024-05-09T04:51:27.885 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2024-05-09T04:51:27.885 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2024-05-09T04:51:27.886 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.06 2024-05-09T04:51:27.886 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 68.el9 2024-05-09T04:51:27.886 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2024-05-09T04:51:27.886 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-09T04:51:27.886 INFO:teuthology.orchestra.run.smithi114.stdout:Group : Unspecified 2024-05-09T04:51:27.886 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 8264254 2024-05-09T04:51:27.886 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2024-05-09T04:51:27.886 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-09T04:51:27.886 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-09T04:51:27.887 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-09T04:51:27.887 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-09T04:51:27.887 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : builder@centos.org 2024-05-09T04:51:27.887 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : CentOS 2024-05-09T04:51:27.887 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-09T04:51:27.887 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2024-05-09T04:51:27.887 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2024-05-09T04:51:27.887 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:51:27.887 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-09T04:51:27.887 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-09T04:51:27.887 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-09T04:51:27.887 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2024-05-09T04:51:27.888 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:51:27.888 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2024-05-09T04:51:27.889 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-09T04:51:27.889 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 5.14.0-444.el9.x86_64 2024-05-09T04:51:27.889 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-09T04:51:28.490 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2024-05-09T04:51:29.834 INFO:teuthology.orchestra.run.smithi114.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-09T04:51:29.876 INFO:teuthology.orchestra.run.smithi114.stderr:done 2024-05-09T04:51:29.878 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-05-09T04:51:29.892 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.ggdzna6d3X 2024-05-09T04:51:29.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ggdzna6d3X 2024-05-09T04:51:29.958 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.ggdzna6d3X 2024-05-09T04:51:30.092 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.ggdzna6d3X is 6KB 2024-05-09T04:51:30.140 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.ggdzna6d3X 2024-05-09T04:51:30.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2024-05-09T04:51:30.217 INFO:teuthology.orchestra.run.smithi114.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-09T04:51:30.217 INFO:teuthology.orchestra.run.smithi114.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-09T04:51:30.217 INFO:teuthology.orchestra.run.smithi114.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-09T04:51:30.217 INFO:teuthology.orchestra.run.smithi114.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-09T04:51:30.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-09T04:51:30.377 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2024-05-09T04:51:41.378 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-05-09T04:51:41.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-05-09T04:51:41.380 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T04:52:00.396 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-05-09T04:52:00.397 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-05-09T04:52:00.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T04:52:34.979 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-05-09T04:52:41.389 DEBUG:teuthology.orchestra.remote:timed out 2024-05-09T04:52:43.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-05-09T04:52:43.983 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T04:52:50.394 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-05-09T04:52:50.395 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T04:52:50.652 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-05-09T04:52:51.133 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-05-09T04:52:51.133 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-444.el9.x86_64"... 2024-05-09T04:52:51.133 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-05-09T04:52:51.148 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-444.el9.x86_64 2024-05-09T04:52:51.148 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-09T04:52:51.148 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-09T04:52:51.149 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-09T04:52:52.149 DEBUG:teuthology.parallel:result is None 2024-05-09T04:53:02.370 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-05-09T04:53:14.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-05-09T04:53:14.372 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T04:53:14.793 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-05-09T04:53:15.283 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-05-09T04:53:15.284 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "5.14.0-444.el9.x86_64"... 2024-05-09T04:53:15.284 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-05-09T04:53:15.298 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-444.el9.x86_64 2024-05-09T04:53:15.298 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-09T04:53:15.299 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-09T04:53:15.299 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-09T04:53:16.299 DEBUG:teuthology.parallel:result is None 2024-05-09T04:53:16.300 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-09T04:53:16.308 INFO:teuthology.task.internal:Creating test directory... 2024-05-09T04:53:16.308 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T04:53:16.312 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T04:53:16.329 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-09T04:53:16.340 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-09T04:53:16.370 INFO:teuthology.task.internal:Creating archive directory... 2024-05-09T04:53:16.370 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T04:53:16.373 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T04:53:16.405 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-09T04:53:16.417 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-09T04:53:16.417 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T04:53:16.443 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T04:53:16.541 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T04:53:16.542 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T04:53:16.570 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T04:53:16.590 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T04:53:16.592 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-09T04:53:16.604 INFO:teuthology.task.internal:Configuring sudo... 2024-05-09T04:53:16.604 DEBUG:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T04:53:16.606 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T04:53:16.634 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-09T04:53:16.644 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-09T04:53:16.644 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T04:53:16.677 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T04:53:16.692 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T04:53:16.778 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T04:53:16.824 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T04:53:16.907 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T04:53:16.950 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:53:16.950 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T04:53:17.017 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T04:53:17.059 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T04:53:17.116 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T04:53:17.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T04:53:17.239 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:53:17.240 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T04:53:17.325 DEBUG:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2024-05-09T04:53:17.328 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-05-09T04:53:17.377 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T04:53:17.407 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T04:53:17.885 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-09T04:53:17.892 INFO:teuthology.task.internal:Starting timer... 2024-05-09T04:53:17.892 INFO:teuthology.run_tasks:Running task pcp... 2024-05-09T04:53:17.928 INFO:teuthology.run_tasks:Running task selinux... 2024-05-09T04:53:17.962 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-05-09T04:53:17.963 DEBUG:teuthology.orchestra.run.smithi060:> sudo service auditd rotate 2024-05-09T04:53:18.038 INFO:teuthology.orchestra.run.smithi060.stdout:Rotating logs: 2024-05-09T04:53:18.041 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-05-09T04:53:18.116 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-05-09T04:53:18.118 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-09T04:53:18.118 DEBUG:teuthology.orchestra.run.smithi060:> /usr/sbin/getenforce 2024-05-09T04:53:18.145 INFO:teuthology.orchestra.run.smithi060.stdout:Permissive 2024-05-09T04:53:18.146 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-05-09T04:53:18.171 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-05-09T04:53:18.172 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi060.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi114.front.sepia.ceph.com': 'permissive'} 2024-05-09T04:53:18.172 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T04:53:18.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T04:53:18.216 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T04:53:18.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T04:53:18.244 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-09T04:53:18.244 DEBUG:teuthology.orchestra.run.smithi060:> sudo /usr/sbin/setenforce permissive 2024-05-09T04:53:18.284 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-05-09T04:53:18.340 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-09T04:53:18.350 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-05-09T04:53:18.355 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-05-09T04:53:18.480 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-09T04:53:18.495 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-09T04:53:18.496 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi060.front.sepia.ceph.com,smithi114.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-09T04:57:18.534 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi060.front.sepia.ceph.com'), Remote(name='ubuntu@smithi114.front.sepia.ceph.com')] 2024-05-09T04:57:18.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-05-09T04:57:18.537 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T04:57:18.617 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-05-09T04:57:18.707 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-05-09T04:57:18.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-05-09T04:57:18.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T04:57:18.782 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-05-09T04:57:18.864 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-05-09T04:57:18.864 INFO:teuthology.run_tasks:Running task clock... 2024-05-09T04:57:18.874 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-09T04:57:18.874 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T04:57:18.875 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T04:57:18.877 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T04:57:18.878 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T04:57:18.907 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T04:57:18.922 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T04:57:18.935 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T04:57:18.948 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: ntpd: command not found 2024-05-09T04:57:18.950 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T04:57:18.960 INFO:teuthology.orchestra.run.smithi060.stdout:506 Cannot talk to daemon 2024-05-09T04:57:18.975 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T04:57:18.975 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-05-09T04:57:18.986 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-05-09T04:57:18.989 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T04:57:19.000 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T04:57:19.014 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T04:57:19.049 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-05-09T04:57:19.052 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T04:57:19.053 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-05-09T04:57:19.053 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T04:57:19.053 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T04:57:19.053 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T04:57:19.053 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T04:57:19.065 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-05-09T04:57:19.069 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T04:57:19.069 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-05-09T04:57:19.069 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T04:57:19.069 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T04:57:19.069 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T04:57:19.069 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T04:57:19.069 INFO:teuthology.run_tasks:Running task install... 2024-05-09T04:57:19.077 DEBUG:teuthology.task.install:project ceph 2024-05-09T04:57:19.077 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '165a15e51d08073b5317c443415b0d8a7141b1ab'}} 2024-05-09T04:57:19.078 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '165a15e51d08073b5317c443415b0d8a7141b1ab'} 2024-05-09T04:57:19.078 INFO:teuthology.task.install:Using flavor: default 2024-05-09T04:57:19.083 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-09T04:57:19.083 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-09T04:57:19.084 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '165a15e51d08073b5317c443415b0d8a7141b1ab', 'tag': None, 'wait_for_package': False} 2024-05-09T04:57:19.084 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=165a15e51d08073b5317c443415b0d8a7141b1ab 2024-05-09T04:57:19.086 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '165a15e51d08073b5317c443415b0d8a7141b1ab', 'tag': None, 'wait_for_package': False} 2024-05-09T04:57:19.086 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=165a15e51d08073b5317c443415b0d8a7141b1ab 2024-05-09T04:57:19.236 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-63259-3/165a15e51d08073b5317c443415b0d8a7141b1ab/centos/9/flavors/default/ 2024-05-09T04:57:19.236 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3568.g165a15e5 2024-05-09T04:57:19.241 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-63259-3/165a15e51d08073b5317c443415b0d8a7141b1ab/centos/9/flavors/default/ 2024-05-09T04:57:19.241 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3568.g165a15e5 2024-05-09T04:57:19.349 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-63259-3/165a15e51d08073b5317c443415b0d8a7141b1ab/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-63259-3/165a15e51d08073b5317c443415b0d8a7141b1ab/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-63259-3/165a15e51d08073b5317c443415b0d8a7141b1ab/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T04:57:19.349 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:57:19.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T04:57:19.375 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T04:57:19.375 DEBUG:teuthology.orchestra.run.smithi114:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/165a15e51d08073b5317c443415b0d8a7141b1ab/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T04:57:19.413 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-63259-3/165a15e51d08073b5317c443415b0d8a7141b1ab/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-63259-3/165a15e51d08073b5317c443415b0d8a7141b1ab/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-63259-3/165a15e51d08073b5317c443415b0d8a7141b1ab/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T04:57:19.413 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:57:19.413 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T04:57:19.440 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T04:57:19.441 DEBUG:teuthology.orchestra.run.smithi060:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/165a15e51d08073b5317c443415b0d8a7141b1ab/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T04:57:19.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T04:57:19.507 DEBUG:teuthology.orchestra.run.smithi060:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T04:57:19.517 DEBUG:teuthology.orchestra.run.smithi114:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T04:57:19.582 INFO:teuthology.orchestra.run.smithi114.stdout:check_obsoletes = 1 2024-05-09T04:57:19.584 DEBUG:teuthology.orchestra.run.smithi060:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T04:57:19.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum clean all 2024-05-09T04:57:19.649 INFO:teuthology.orchestra.run.smithi060.stdout:check_obsoletes = 1 2024-05-09T04:57:19.650 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum clean all 2024-05-09T04:57:19.817 INFO:teuthology.orchestra.run.smithi114.stdout:54 files removed 2024-05-09T04:57:19.844 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T04:57:19.897 INFO:teuthology.orchestra.run.smithi060.stdout:54 files removed 2024-05-09T04:57:19.925 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T04:57:20.512 INFO:teuthology.orchestra.run.smithi114.stdout:ceph packages for x86_64 213 kB/s | 84 kB 00:00 2024-05-09T04:57:20.728 INFO:teuthology.orchestra.run.smithi060.stdout:ceph packages for x86_64 162 kB/s | 84 kB 00:00 2024-05-09T04:57:20.946 INFO:teuthology.orchestra.run.smithi114.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-05-09T04:57:21.329 INFO:teuthology.orchestra.run.smithi060.stdout:ceph noarch packages 21 kB/s | 12 kB 00:00 2024-05-09T04:57:21.388 INFO:teuthology.orchestra.run.smithi114.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-05-09T04:57:21.771 INFO:teuthology.orchestra.run.smithi060.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-05-09T04:57:21.997 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.1 MB 00:00 2024-05-09T04:57:22.971 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 7.0 MB/s | 8.1 MB 00:01 2024-05-09T04:57:24.890 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-05-09T04:57:25.847 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-05-09T04:57:30.299 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.2 MB 00:01 2024-05-09T04:57:31.432 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.2 MB 00:01 2024-05-09T04:57:31.933 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 16 kB 00:00 2024-05-09T04:57:33.024 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 16 kB 00:00 2024-05-09T04:57:34.295 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-05-09T04:57:39.709 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-05-09T04:57:42.154 INFO:teuthology.orchestra.run.smithi060.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T04:57:42.155 INFO:teuthology.orchestra.run.smithi060.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T04:57:42.167 INFO:teuthology.orchestra.run.smithi060.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T04:57:42.167 INFO:teuthology.orchestra.run.smithi060.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T04:57:42.168 INFO:teuthology.orchestra.run.smithi060.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T04:57:42.169 INFO:teuthology.orchestra.run.smithi060.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T04:57:42.171 INFO:teuthology.orchestra.run.smithi060.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T04:57:42.173 INFO:teuthology.orchestra.run.smithi060.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T04:57:42.174 INFO:teuthology.orchestra.run.smithi060.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T04:57:42.240 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-05-09T04:57:42.254 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================= 2024-05-09T04:57:42.254 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-05-09T04:57:42.254 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================= 2024-05-09T04:57:42.254 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-05-09T04:57:42.254 INFO:teuthology.orchestra.run.smithi060.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T04:57:42.255 INFO:teuthology.orchestra.run.smithi060.stdout: ceph x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 6.6 k 2024-05-09T04:57:42.255 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 5.4 M 2024-05-09T04:57:42.255 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 890 k 2024-05-09T04:57:42.255 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 145 k 2024-05-09T04:57:42.255 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 1.5 M 2024-05-09T04:57:42.255 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 142 k 2024-05-09T04:57:42.255 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 3.5 M 2024-05-09T04:57:42.255 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 7.4 M 2024-05-09T04:57:42.255 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 49 k 2024-05-09T04:57:42.255 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 11 M 2024-05-09T04:57:42.255 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 49 M 2024-05-09T04:57:42.255 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 273 k 2024-05-09T04:57:42.255 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 764 k 2024-05-09T04:57:42.256 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-mirror x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 225 k 2024-05-09T04:57:42.256 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-shell x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 27 k 2024-05-09T04:57:42.256 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-top noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 18 k 2024-05-09T04:57:42.256 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T04:57:42.256 INFO:teuthology.orchestra.run.smithi060.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T04:57:42.256 INFO:teuthology.orchestra.run.smithi060.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T04:57:42.256 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 32 k 2024-05-09T04:57:42.256 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2 x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 747 k 2024-05-09T04:57:42.256 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 127 k 2024-05-09T04:57:42.256 INFO:teuthology.orchestra.run.smithi060.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T04:57:42.256 INFO:teuthology.orchestra.run.smithi060.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T04:57:42.256 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T04:57:42.257 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T04:57:42.257 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T04:57:42.257 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T04:57:42.257 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 162 k 2024-05-09T04:57:42.257 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 321 k 2024-05-09T04:57:42.257 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 299 k 2024-05-09T04:57:42.257 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 100 k 2024-05-09T04:57:42.257 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 85 k 2024-05-09T04:57:42.257 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 3.1 M 2024-05-09T04:57:42.257 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 171 k 2024-05-09T04:57:42.257 INFO:teuthology.orchestra.run.smithi060.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T04:57:42.257 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrading: 2024-05-09T04:57:42.258 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T04:57:42.258 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T04:57:42.258 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T04:57:42.258 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T04:57:42.258 INFO:teuthology.orchestra.run.smithi060.stdout: librados2 x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 3.4 M 2024-05-09T04:57:42.258 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1 x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 3.1 M 2024-05-09T04:57:42.258 INFO:teuthology.orchestra.run.smithi060.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T04:57:42.258 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T04:57:42.258 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-05-09T04:57:42.258 INFO:teuthology.orchestra.run.smithi060.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T04:57:42.258 INFO:teuthology.orchestra.run.smithi060.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T04:57:42.258 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T04:57:42.258 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 21 M 2024-05-09T04:57:42.258 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 34 k 2024-05-09T04:57:42.259 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 2.4 M 2024-05-09T04:57:42.259 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 274 k 2024-05-09T04:57:42.259 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 4.7 M 2024-05-09T04:57:42.259 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 17 M 2024-05-09T04:57:42.259 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 16 k 2024-05-09T04:57:42.259 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 25 k 2024-05-09T04:57:42.259 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T04:57:42.259 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T04:57:42.259 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T04:57:42.259 INFO:teuthology.orchestra.run.smithi060.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T04:57:42.259 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T04:57:42.259 INFO:teuthology.orchestra.run.smithi060.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T04:57:42.259 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T04:57:42.259 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T04:57:42.260 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 163 k 2024-05-09T04:57:42.260 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T04:57:42.260 INFO:teuthology.orchestra.run.smithi060.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T04:57:42.260 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T04:57:42.260 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T04:57:42.260 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T04:57:42.260 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1 x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 500 k 2024-05-09T04:57:42.260 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T04:57:42.260 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2 x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 5.4 M 2024-05-09T04:57:42.260 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T04:57:42.260 INFO:teuthology.orchestra.run.smithi060.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T04:57:42.261 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T04:57:42.261 INFO:teuthology.orchestra.run.smithi060.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T04:57:42.261 INFO:teuthology.orchestra.run.smithi060.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T04:57:42.261 INFO:teuthology.orchestra.run.smithi060.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T04:57:42.261 INFO:teuthology.orchestra.run.smithi060.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T04:57:42.261 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T04:57:42.261 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T04:57:42.261 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T04:57:42.261 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T04:57:42.261 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T04:57:42.261 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T04:57:42.261 INFO:teuthology.orchestra.run.smithi060.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T04:57:42.261 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T04:57:42.261 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T04:57:42.262 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T04:57:42.262 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 45 k 2024-05-09T04:57:42.262 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 135 k 2024-05-09T04:57:42.262 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T04:57:42.262 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T04:57:42.262 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T04:57:42.262 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T04:57:42.262 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T04:57:42.262 INFO:teuthology.orchestra.run.smithi060.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T04:57:42.262 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T04:57:42.262 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T04:57:42.262 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T04:57:42.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T04:57:42.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T04:57:42.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T04:57:42.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T04:57:42.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T04:57:42.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T04:57:42.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T04:57:42.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T04:57:42.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T04:57:42.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T04:57:42.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T04:57:42.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T04:57:42.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T04:57:42.264 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T04:57:42.264 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T04:57:42.264 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T04:57:42.264 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T04:57:42.264 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T04:57:42.264 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T04:57:42.264 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T04:57:42.264 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T04:57:42.264 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T04:57:42.264 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T04:57:42.265 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T04:57:42.265 INFO:teuthology.orchestra.run.smithi060.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T04:57:42.265 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T04:57:42.265 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T04:57:42.265 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T04:57:42.265 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T04:57:42.265 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T04:57:42.265 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T04:57:42.265 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T04:57:42.265 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T04:57:42.265 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T04:57:42.265 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T04:57:42.266 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T04:57:42.266 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T04:57:42.266 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T04:57:42.266 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T04:57:42.266 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T04:57:42.266 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T04:57:42.266 INFO:teuthology.orchestra.run.smithi060.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T04:57:42.266 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T04:57:42.266 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T04:57:42.266 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T04:57:42.266 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T04:57:42.266 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T04:57:42.267 INFO:teuthology.orchestra.run.smithi060.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T04:57:42.267 INFO:teuthology.orchestra.run.smithi060.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T04:57:42.267 INFO:teuthology.orchestra.run.smithi060.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T04:57:42.267 INFO:teuthology.orchestra.run.smithi060.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T04:57:42.267 INFO:teuthology.orchestra.run.smithi060.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T04:57:42.267 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T04:57:42.267 INFO:teuthology.orchestra.run.smithi060.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T04:57:42.267 INFO:teuthology.orchestra.run.smithi060.stdout:Installing weak dependencies: 2024-05-09T04:57:42.267 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T04:57:42.267 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:57:42.267 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-05-09T04:57:42.268 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================= 2024-05-09T04:57:42.268 INFO:teuthology.orchestra.run.smithi060.stdout:Install 148 Packages 2024-05-09T04:57:42.268 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrade 8 Packages 2024-05-09T04:57:42.268 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:57:42.268 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 226 M 2024-05-09T04:57:42.268 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-05-09T04:57:42.953 INFO:teuthology.orchestra.run.smithi060.stdout:(1/156): ceph-19.0.0-3568.g165a15e5.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-05-09T04:57:43.312 INFO:teuthology.orchestra.run.smithi060.stdout:(2/156): ceph-fuse-19.0.0-3568.g165a15e5.el9.x8 2.4 MB/s | 890 kB 00:00 2024-05-09T04:57:43.412 INFO:teuthology.orchestra.run.smithi060.stdout:(3/156): ceph-immutable-object-cache-19.0.0-356 1.4 MB/s | 145 kB 00:00 2024-05-09T04:57:44.062 INFO:teuthology.orchestra.run.smithi060.stdout:(4/156): ceph-common-19.0.0-3568.g165a15e5.el9. 16 MB/s | 21 MB 00:01 2024-05-09T04:57:44.112 INFO:teuthology.orchestra.run.smithi060.stdout:(5/156): ceph-mds-19.0.0-3568.g165a15e5.el9.x86 3.4 MB/s | 2.4 MB 00:00 2024-05-09T04:57:44.171 INFO:teuthology.orchestra.run.smithi060.stdout:(6/156): ceph-mgr-19.0.0-3568.g165a15e5.el9.x86 14 MB/s | 1.5 MB 00:00 2024-05-09T04:57:45.038 INFO:teuthology.orchestra.run.smithi060.stdout:(7/156): ceph-osd-19.0.0-3568.g165a15e5.el9.x86 19 MB/s | 17 MB 00:00 2024-05-09T04:57:45.597 INFO:teuthology.orchestra.run.smithi060.stdout:(8/156): ceph-radosgw-19.0.0-3568.g165a15e5.el9 19 MB/s | 11 MB 00:00 2024-05-09T04:57:45.664 INFO:teuthology.orchestra.run.smithi060.stdout:(9/156): ceph-mon-19.0.0-3568.g165a15e5.el9.x86 3.0 MB/s | 4.7 MB 00:01 2024-05-09T04:57:45.689 INFO:teuthology.orchestra.run.smithi060.stdout:(10/156): ceph-selinux-19.0.0-3568.g165a15e5.el 274 kB/s | 25 kB 00:00 2024-05-09T04:57:45.747 INFO:teuthology.orchestra.run.smithi060.stdout:(11/156): cephfs-mirror-19.0.0-3568.g165a15e5.e 3.8 MB/s | 225 kB 00:00 2024-05-09T04:57:45.797 INFO:teuthology.orchestra.run.smithi060.stdout:(12/156): cephfs-shell-19.0.0-3568.g165a15e5.el 541 kB/s | 27 kB 00:00 2024-05-09T04:57:45.840 INFO:teuthology.orchestra.run.smithi060.stdout:(13/156): libcephfs-devel-19.0.0-3568.g165a15e5 761 kB/s | 32 kB 00:00 2024-05-09T04:57:45.924 INFO:teuthology.orchestra.run.smithi060.stdout:(14/156): libcephfs2-19.0.0-3568.g165a15e5.el9. 8.7 MB/s | 747 kB 00:00 2024-05-09T04:57:45.974 INFO:teuthology.orchestra.run.smithi060.stdout:(15/156): libcephsqlite-19.0.0-3568.g165a15e5.e 3.2 MB/s | 163 kB 00:00 2024-05-09T04:57:46.024 INFO:teuthology.orchestra.run.smithi060.stdout:(16/156): librados-devel-19.0.0-3568.g165a15e5. 2.5 MB/s | 127 kB 00:00 2024-05-09T04:57:46.091 INFO:teuthology.orchestra.run.smithi060.stdout:(17/156): libradosstriper1-19.0.0-3568.g165a15e 7.4 MB/s | 500 kB 00:00 2024-05-09T04:57:46.408 INFO:teuthology.orchestra.run.smithi060.stdout:(18/156): librgw2-19.0.0-3568.g165a15e5.el9.x86 17 MB/s | 5.4 MB 00:00 2024-05-09T04:57:46.475 INFO:teuthology.orchestra.run.smithi060.stdout:(19/156): ceph-base-19.0.0-3568.g165a15e5.el9.x 1.5 MB/s | 5.4 MB 00:03 2024-05-09T04:57:46.500 INFO:teuthology.orchestra.run.smithi060.stdout:(20/156): python3-ceph-argparse-19.0.0-3568.g16 491 kB/s | 45 kB 00:00 2024-05-09T04:57:46.551 INFO:teuthology.orchestra.run.smithi060.stdout:(21/156): python3-cephfs-19.0.0-3568.g165a15e5. 3.2 MB/s | 162 kB 00:00 2024-05-09T04:57:46.584 INFO:teuthology.orchestra.run.smithi060.stdout:(22/156): python3-ceph-common-19.0.0-3568.g165a 1.2 MB/s | 135 kB 00:00 2024-05-09T04:57:46.609 INFO:teuthology.orchestra.run.smithi060.stdout:(23/156): python3-rados-19.0.0-3568.g165a15e5.e 5.4 MB/s | 321 kB 00:00 2024-05-09T04:57:46.660 INFO:teuthology.orchestra.run.smithi060.stdout:(24/156): python3-rgw-19.0.0-3568.g165a15e5.el9 1.9 MB/s | 100 kB 00:00 2024-05-09T04:57:46.710 INFO:teuthology.orchestra.run.smithi060.stdout:(25/156): rbd-fuse-19.0.0-3568.g165a15e5.el9.x8 1.7 MB/s | 85 kB 00:00 2024-05-09T04:57:46.735 INFO:teuthology.orchestra.run.smithi060.stdout:(26/156): python3-rbd-19.0.0-3568.g165a15e5.el9 1.9 MB/s | 299 kB 00:00 2024-05-09T04:57:46.905 INFO:teuthology.orchestra.run.smithi060.stdout:(27/156): rbd-mirror-19.0.0-3568.g165a15e5.el9. 16 MB/s | 3.1 MB 00:00 2024-05-09T04:57:46.928 INFO:teuthology.orchestra.run.smithi060.stdout:(28/156): rbd-nbd-19.0.0-3568.g165a15e5.el9.x86 891 kB/s | 171 kB 00:00 2024-05-09T04:57:46.953 INFO:teuthology.orchestra.run.smithi060.stdout:(29/156): ceph-grafana-dashboards-19.0.0-3568.g 682 kB/s | 34 kB 00:00 2024-05-09T04:57:47.070 INFO:teuthology.orchestra.run.smithi060.stdout:(30/156): ceph-mgr-cephadm-19.0.0-3568.g165a15e 1.0 MB/s | 142 kB 00:00 2024-05-09T04:57:47.170 INFO:teuthology.orchestra.run.smithi060.stdout:(31/156): ceph-mgr-dashboard-19.0.0-3568.g165a1 16 MB/s | 3.5 MB 00:00 2024-05-09T04:57:47.221 INFO:teuthology.orchestra.run.smithi060.stdout:(32/156): ceph-mgr-modules-core-19.0.0-3568.g16 5.4 MB/s | 274 kB 00:00 2024-05-09T04:57:47.271 INFO:teuthology.orchestra.run.smithi060.stdout:(33/156): ceph-mgr-rook-19.0.0-3568.g165a15e5.e 988 kB/s | 49 kB 00:00 2024-05-09T04:57:47.321 INFO:teuthology.orchestra.run.smithi060.stdout:(34/156): ceph-prometheus-alerts-19.0.0-3568.g1 329 kB/s | 16 kB 00:00 2024-05-09T04:57:47.372 INFO:teuthology.orchestra.run.smithi060.stdout:(35/156): ceph-volume-19.0.0-3568.g165a15e5.el9 5.3 MB/s | 273 kB 00:00 2024-05-09T04:57:47.447 INFO:teuthology.orchestra.run.smithi060.stdout:(36/156): cephadm-19.0.0-3568.g165a15e5.el9.noa 10 MB/s | 764 kB 00:00 2024-05-09T04:57:47.497 INFO:teuthology.orchestra.run.smithi060.stdout:(37/156): cephfs-top-19.0.0-3568.g165a15e5.el9. 364 kB/s | 18 kB 00:00 2024-05-09T04:57:47.914 INFO:teuthology.orchestra.run.smithi060.stdout:(38/156): fuse-2.9.9-15.el9.x86_64.rpm 193 kB/s | 80 kB 00:00 2024-05-09T04:57:48.340 INFO:teuthology.orchestra.run.smithi060.stdout:(39/156): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-05-09T04:57:48.448 INFO:teuthology.orchestra.run.smithi060.stdout:(40/156): libquadmath-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 195 kB 00:00 2024-05-09T04:57:48.557 INFO:teuthology.orchestra.run.smithi060.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 253 kB 00:00 2024-05-09T04:57:48.832 INFO:teuthology.orchestra.run.smithi060.stdout:(42/156): python3-cryptography-36.0.1-4.el9.x86 4.5 MB/s | 1.2 MB 00:00 2024-05-09T04:57:48.941 INFO:teuthology.orchestra.run.smithi060.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 985 kB/s | 106 kB 00:00 2024-05-09T04:57:49.253 INFO:teuthology.orchestra.run.smithi060.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-05-09T04:57:49.253 INFO:teuthology.orchestra.run.smithi060.stdout:(45/156): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-05-09T04:57:49.267 INFO:teuthology.orchestra.run.smithi060.stdout:(46/156): python3-urllib3-1.26.5-5.el9.noarch.r 1.9 MB/s | 215 kB 00:00 2024-05-09T04:57:49.409 INFO:teuthology.orchestra.run.smithi060.stdout:(47/156): ceph-mgr-diskprediction-local-19.0.0- 3.2 MB/s | 7.4 MB 00:02 2024-05-09T04:57:50.259 INFO:teuthology.orchestra.run.smithi060.stdout:(48/156): automake-1.16.2-8.el9.noarch.rpm 695 kB/s | 689 kB 00:00 2024-05-09T04:57:50.368 INFO:teuthology.orchestra.run.smithi060.stdout:(49/156): bison-3.7.4-5.el9.x86_64.rpm 983 kB/s | 943 kB 00:00 2024-05-09T04:57:50.975 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:18 2024-05-09T04:57:51.160 INFO:teuthology.orchestra.run.smithi060.stdout:(50/156): ceph-test-19.0.0-3568.g165a15e5.el9.x 8.9 MB/s | 49 MB 00:05 2024-05-09T04:57:51.585 INFO:teuthology.orchestra.run.smithi060.stdout:(51/156): elfutils-libelf-devel-0.190-2.el9.x86 18 kB/s | 21 kB 00:01 2024-05-09T04:57:51.610 INFO:teuthology.orchestra.run.smithi060.stdout:(52/156): boost-program-options-1.75.0-8.el9.x8 79 kB/s | 107 kB 00:01 2024-05-09T04:57:52.144 INFO:teuthology.orchestra.run.smithi060.stdout:(53/156): flex-2.6.4-9.el9.x86_64.rpm 329 kB/s | 313 kB 00:00 2024-05-09T04:57:52.177 INFO:teuthology.orchestra.run.smithi060.stdout:(54/156): flexiblas-3.0.4-8.el9.x86_64.rpm 55 kB/s | 33 kB 00:00 2024-05-09T04:57:52.677 INFO:teuthology.orchestra.run.smithi060.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 34 kB/s | 18 kB 00:00 2024-05-09T04:57:52.703 INFO:teuthology.orchestra.run.smithi060.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 21 kB/s | 11 kB 00:00 2024-05-09T04:57:52.762 INFO:teuthology.orchestra.run.smithi060.stdout:(57/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.6 MB/s | 3.0 MB 00:01 2024-05-09T04:57:53.345 INFO:teuthology.orchestra.run.smithi060.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 70 kB/s | 45 kB 00:00 2024-05-09T04:57:53.454 INFO:teuthology.orchestra.run.smithi060.stdout:(59/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 207 kB/s | 160 kB 00:00 2024-05-09T04:57:53.737 INFO:teuthology.orchestra.run.smithi060.stdout:(60/156): librdkafka-1.6.1-102.el9.x86_64.rpm 679 kB/s | 662 kB 00:00 2024-05-09T04:57:54.029 INFO:teuthology.orchestra.run.smithi060.stdout:(61/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 44 kB/s | 25 kB 00:00 2024-05-09T04:57:54.248 INFO:teuthology.orchestra.run.smithi060.stdout:(62/156): libtool-2.4.6-45.el9.x86_64.rpm 647 kB/s | 582 kB 00:00 2024-05-09T04:57:54.363 INFO:teuthology.orchestra.run.smithi060.stdout:(63/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 76 kB/s | 47 kB 00:00 2024-05-09T04:57:54.888 INFO:teuthology.orchestra.run.smithi060.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 341 kB/s | 292 kB 00:00 2024-05-09T04:57:54.972 INFO:teuthology.orchestra.run.smithi060.stdout:(65/156): openblas-0.3.21-2.el9.x86_64.rpm 58 kB/s | 35 kB 00:00 2024-05-09T04:57:55.039 INFO:teuthology.orchestra.run.smithi060.stdout:(66/156): lua-5.4.4-4.el9.x86_64.rpm 238 kB/s | 188 kB 00:00 2024-05-09T04:57:56.073 INFO:teuthology.orchestra.run.smithi060.stdout:(67/156): openblas-openmp-0.3.21-2.el9.x86_64.r 4.0 MB/s | 4.7 MB 00:01 2024-05-09T04:57:56.115 INFO:teuthology.orchestra.run.smithi060.stdout:(68/156): postgresql-13.14-1.el9.x86_64.rpm 1.5 MB/s | 1.6 MB 00:01 2024-05-09T04:57:56.150 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-05-09T04:57:56.182 INFO:teuthology.orchestra.run.smithi060.stdout:(69/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 3.7 MB/s | 4.4 MB 00:01 2024-05-09T04:57:56.907 INFO:teuthology.orchestra.run.smithi060.stdout:(70/156): postgresql-private-libs-13.14-1.el9.x 168 kB/s | 133 kB 00:00 2024-05-09T04:57:57.057 INFO:teuthology.orchestra.run.smithi060.stdout:(71/156): postgresql-contrib-13.14-1.el9.x86_64 926 kB/s | 911 kB 00:00 2024-05-09T04:57:57.441 INFO:teuthology.orchestra.run.smithi060.stdout:(72/156): postgresql-server-13.14-1.el9.x86_64. 4.6 MB/s | 5.8 MB 00:01 2024-05-09T04:57:57.808 INFO:teuthology.orchestra.run.smithi060.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 117 kB/s | 87 kB 00:00 2024-05-09T04:57:57.925 INFO:teuthology.orchestra.run.smithi060.stdout:(74/156): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:01 2024-05-09T04:57:58.461 INFO:teuthology.orchestra.run.smithi114.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T04:57:58.462 INFO:teuthology.orchestra.run.smithi114.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T04:57:58.473 INFO:teuthology.orchestra.run.smithi114.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T04:57:58.474 INFO:teuthology.orchestra.run.smithi114.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T04:57:58.475 INFO:teuthology.orchestra.run.smithi114.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T04:57:58.476 INFO:teuthology.orchestra.run.smithi114.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T04:57:58.477 INFO:teuthology.orchestra.run.smithi114.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T04:57:58.479 INFO:teuthology.orchestra.run.smithi114.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T04:57:58.480 INFO:teuthology.orchestra.run.smithi114.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T04:57:58.543 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-05-09T04:57:58.564 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-05-09T04:57:58.564 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-05-09T04:57:58.564 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-05-09T04:57:58.564 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-05-09T04:57:58.564 INFO:teuthology.orchestra.run.smithi114.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T04:57:58.564 INFO:teuthology.orchestra.run.smithi114.stdout: ceph x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 6.6 k 2024-05-09T04:57:58.564 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 5.4 M 2024-05-09T04:57:58.564 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 890 k 2024-05-09T04:57:58.564 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 145 k 2024-05-09T04:57:58.564 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 1.5 M 2024-05-09T04:57:58.564 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 142 k 2024-05-09T04:57:58.564 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 3.5 M 2024-05-09T04:57:58.565 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 7.4 M 2024-05-09T04:57:58.565 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 49 k 2024-05-09T04:57:58.565 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 11 M 2024-05-09T04:57:58.565 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 49 M 2024-05-09T04:57:58.565 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 273 k 2024-05-09T04:57:58.565 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 764 k 2024-05-09T04:57:58.565 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-mirror x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 225 k 2024-05-09T04:57:58.565 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-shell x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 27 k 2024-05-09T04:57:58.565 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-top noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 18 k 2024-05-09T04:57:58.565 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T04:57:58.565 INFO:teuthology.orchestra.run.smithi114.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T04:57:58.565 INFO:teuthology.orchestra.run.smithi114.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T04:57:58.565 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 32 k 2024-05-09T04:57:58.566 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 747 k 2024-05-09T04:57:58.566 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 127 k 2024-05-09T04:57:58.566 INFO:teuthology.orchestra.run.smithi114.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T04:57:58.566 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T04:57:58.566 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T04:57:58.566 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T04:57:58.566 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T04:57:58.566 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T04:57:58.566 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 162 k 2024-05-09T04:57:58.566 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 321 k 2024-05-09T04:57:58.566 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 299 k 2024-05-09T04:57:58.566 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 100 k 2024-05-09T04:57:58.567 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 85 k 2024-05-09T04:57:58.567 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 3.1 M 2024-05-09T04:57:58.567 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 171 k 2024-05-09T04:57:58.567 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T04:57:58.567 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-05-09T04:57:58.567 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T04:57:58.567 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T04:57:58.567 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T04:57:58.567 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T04:57:58.567 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 3.4 M 2024-05-09T04:57:58.567 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1 x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 3.1 M 2024-05-09T04:57:58.567 INFO:teuthology.orchestra.run.smithi114.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T04:57:58.567 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T04:57:58.568 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-05-09T04:57:58.568 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T04:57:58.568 INFO:teuthology.orchestra.run.smithi114.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T04:57:58.568 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T04:57:58.568 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 21 M 2024-05-09T04:57:58.568 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 34 k 2024-05-09T04:57:58.568 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 2.4 M 2024-05-09T04:57:58.568 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 274 k 2024-05-09T04:57:58.568 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 4.7 M 2024-05-09T04:57:58.568 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 17 M 2024-05-09T04:57:58.569 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3568.g165a15e5.el9 ceph-noarch 16 k 2024-05-09T04:57:58.569 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 25 k 2024-05-09T04:57:58.569 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T04:57:58.569 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T04:57:58.569 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T04:57:58.569 INFO:teuthology.orchestra.run.smithi114.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T04:57:58.569 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T04:57:58.569 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T04:57:58.569 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T04:57:58.569 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T04:57:58.569 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 163 k 2024-05-09T04:57:58.569 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T04:57:58.570 INFO:teuthology.orchestra.run.smithi114.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T04:57:58.570 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T04:57:58.570 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T04:57:58.570 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T04:57:58.570 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 500 k 2024-05-09T04:57:58.570 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T04:57:58.570 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2 x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 5.4 M 2024-05-09T04:57:58.570 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T04:57:58.570 INFO:teuthology.orchestra.run.smithi114.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T04:57:58.570 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T04:57:58.570 INFO:teuthology.orchestra.run.smithi114.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T04:57:58.570 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T04:57:58.571 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T04:57:58.571 INFO:teuthology.orchestra.run.smithi114.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T04:57:58.571 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T04:57:58.571 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T04:57:58.571 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T04:57:58.571 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T04:57:58.571 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T04:57:58.571 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T04:57:58.571 INFO:teuthology.orchestra.run.smithi114.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T04:57:58.571 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T04:57:58.571 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T04:57:58.571 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T04:57:58.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 45 k 2024-05-09T04:57:58.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common x86_64 2:19.0.0-3568.g165a15e5.el9 ceph 135 k 2024-05-09T04:57:58.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T04:57:58.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T04:57:58.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T04:57:58.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T04:57:58.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T04:57:58.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T04:57:58.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T04:57:58.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T04:57:58.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T04:57:58.572 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T04:57:58.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T04:57:58.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T04:57:58.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T04:57:58.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T04:57:58.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T04:57:58.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T04:57:58.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T04:57:58.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T04:57:58.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T04:57:58.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T04:57:58.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T04:57:58.573 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T04:57:58.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T04:57:58.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T04:57:58.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T04:57:58.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T04:57:58.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T04:57:58.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T04:57:58.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T04:57:58.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T04:57:58.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T04:57:58.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T04:57:58.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T04:57:58.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T04:57:58.574 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T04:57:58.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T04:57:58.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T04:57:58.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T04:57:58.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T04:57:58.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T04:57:58.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T04:57:58.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T04:57:58.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T04:57:58.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T04:57:58.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T04:57:58.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T04:57:58.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T04:57:58.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T04:57:58.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T04:57:58.575 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T04:57:58.576 INFO:teuthology.orchestra.run.smithi114.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T04:57:58.576 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T04:57:58.576 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T04:57:58.576 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T04:57:58.576 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T04:57:58.576 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T04:57:58.576 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T04:57:58.576 INFO:teuthology.orchestra.run.smithi114.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T04:57:58.576 INFO:teuthology.orchestra.run.smithi114.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T04:57:58.576 INFO:teuthology.orchestra.run.smithi114.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T04:57:58.576 INFO:teuthology.orchestra.run.smithi114.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T04:57:58.576 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T04:57:58.576 INFO:teuthology.orchestra.run.smithi114.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T04:57:58.576 INFO:teuthology.orchestra.run.smithi114.stdout:Installing weak dependencies: 2024-05-09T04:57:58.577 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T04:57:58.577 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:57:58.577 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-05-09T04:57:58.577 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-05-09T04:57:58.577 INFO:teuthology.orchestra.run.smithi114.stdout:Install 148 Packages 2024-05-09T04:57:58.577 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 8 Packages 2024-05-09T04:57:58.577 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:57:58.577 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 226 M 2024-05-09T04:57:58.577 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-05-09T04:57:58.642 INFO:teuthology.orchestra.run.smithi060.stdout:(75/156): python3-babel-2.9.1-2.el9.noarch.rpm 5.0 MB/s | 6.0 MB 00:01 2024-05-09T04:57:58.667 INFO:teuthology.orchestra.run.smithi060.stdout:(76/156): python3-devel-3.9.18-3.el9.x86_64.rpm 284 kB/s | 244 kB 00:00 2024-05-09T04:57:58.742 INFO:teuthology.orchestra.run.smithi060.stdout:(77/156): python3-jinja2-2.11.3-5.el9.noarch.rp 303 kB/s | 248 kB 00:00 2024-05-09T04:57:59.251 INFO:teuthology.orchestra.run.smithi060.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 42 kB/s | 26 kB 00:00 2024-05-09T04:57:59.276 INFO:teuthology.orchestra.run.smithi060.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 31 kB/s | 19 kB 00:00 2024-05-09T04:57:59.368 INFO:teuthology.orchestra.run.smithi114.stdout:(1/156): ceph-19.0.0-3568.g165a15e5.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-05-09T04:57:59.571 INFO:teuthology.orchestra.run.smithi060.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 209 kB/s | 172 kB 00:00 2024-05-09T04:57:59.596 INFO:teuthology.orchestra.run.smithi114.stdout:(2/156): ceph-fuse-19.0.0-3568.g165a15e5.el9.x8 3.9 MB/s | 890 kB 00:00 2024-05-09T04:57:59.669 INFO:teuthology.orchestra.run.smithi114.stdout:(3/156): ceph-immutable-object-cache-19.0.0-356 1.9 MB/s | 145 kB 00:00 2024-05-09T04:57:59.852 INFO:teuthology.orchestra.run.smithi060.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 58 kB/s | 35 kB 00:00 2024-05-09T04:58:00.061 INFO:teuthology.orchestra.run.smithi114.stdout:(4/156): ceph-mds-19.0.0-3568.g165a15e5.el9.x86 6.0 MB/s | 2.4 MB 00:00 2024-05-09T04:58:00.327 INFO:teuthology.orchestra.run.smithi114.stdout:(5/156): ceph-mgr-19.0.0-3568.g165a15e5.el9.x86 5.7 MB/s | 1.5 MB 00:00 2024-05-09T04:58:00.377 INFO:teuthology.orchestra.run.smithi060.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 310 kB/s | 250 kB 00:00 2024-05-09T04:58:00.469 INFO:teuthology.orchestra.run.smithi060.stdout:(83/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.3 MB/s | 5.1 MB 00:01 2024-05-09T04:58:00.686 INFO:teuthology.orchestra.run.smithi060.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 266 kB/s | 222 kB 00:00 2024-05-09T04:58:00.853 INFO:teuthology.orchestra.run.smithi114.stdout:(6/156): ceph-common-19.0.0-3568.g165a15e5.el9. 13 MB/s | 21 MB 00:01 2024-05-09T04:58:00.928 INFO:teuthology.orchestra.run.smithi114.stdout:(7/156): ceph-base-19.0.0-3568.g165a15e5.el9.x8 3.1 MB/s | 5.4 MB 00:01 2024-05-09T04:58:01.003 INFO:teuthology.orchestra.run.smithi114.stdout:(8/156): ceph-mon-19.0.0-3568.g165a15e5.el9.x86 7.0 MB/s | 4.7 MB 00:00 2024-05-09T04:58:01.078 INFO:teuthology.orchestra.run.smithi060.stdout:(85/156): python3-packaging-20.9-5.el9.noarch.r 110 kB/s | 77 kB 00:00 2024-05-09T04:58:01.078 INFO:teuthology.orchestra.run.smithi114.stdout:(9/156): ceph-selinux-19.0.0-3568.g165a15e5.el9 336 kB/s | 25 kB 00:00 2024-05-09T04:58:01.103 INFO:teuthology.orchestra.run.smithi060.stdout:(86/156): python3-prettytable-0.7.2-27.el9.noar 67 kB/s | 42 kB 00:00 2024-05-09T04:58:01.520 INFO:teuthology.orchestra.run.smithi060.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 323 kB/s | 269 kB 00:00 2024-05-09T04:58:01.845 INFO:teuthology.orchestra.run.smithi060.stdout:(88/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 207 kB/s | 159 kB 00:00 2024-05-09T04:58:01.970 INFO:teuthology.orchestra.run.smithi060.stdout:(89/156): python3-pyasn1-modules-0.4.8-6.el9.no 322 kB/s | 279 kB 00:00 2024-05-09T04:58:02.004 INFO:teuthology.orchestra.run.smithi114.stdout:(10/156): ceph-osd-19.0.0-3568.g165a15e5.el9.x8 15 MB/s | 17 MB 00:01 2024-05-09T04:58:02.079 INFO:teuthology.orchestra.run.smithi114.stdout:(11/156): cephfs-mirror-19.0.0-3568.g165a15e5.e 2.9 MB/s | 225 kB 00:00 2024-05-09T04:58:02.146 INFO:teuthology.orchestra.run.smithi114.stdout:(12/156): cephfs-shell-19.0.0-3568.g165a15e5.el 405 kB/s | 27 kB 00:00 2024-05-09T04:58:02.212 INFO:teuthology.orchestra.run.smithi060.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 74 kB/s | 51 kB 00:00 2024-05-09T04:58:02.221 INFO:teuthology.orchestra.run.smithi114.stdout:(13/156): libcephfs-devel-19.0.0-3568.g165a15e5 421 kB/s | 32 kB 00:00 2024-05-09T04:58:02.313 INFO:teuthology.orchestra.run.smithi114.stdout:(14/156): libcephfs2-19.0.0-3568.g165a15e5.el9. 8.0 MB/s | 747 kB 00:00 2024-05-09T04:58:02.380 INFO:teuthology.orchestra.run.smithi114.stdout:(15/156): libcephsqlite-19.0.0-3568.g165a15e5.e 2.4 MB/s | 163 kB 00:00 2024-05-09T04:58:02.447 INFO:teuthology.orchestra.run.smithi114.stdout:(16/156): librados-devel-19.0.0-3568.g165a15e5. 1.9 MB/s | 127 kB 00:00 2024-05-09T04:58:02.512 INFO:teuthology.orchestra.run.smithi060.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 80 kB/s | 54 kB 00:00 2024-05-09T04:58:02.522 INFO:teuthology.orchestra.run.smithi114.stdout:(17/156): libradosstriper1-19.0.0-3568.g165a15e 6.5 MB/s | 500 kB 00:00 2024-05-09T04:58:02.931 INFO:teuthology.orchestra.run.smithi114.stdout:(18/156): ceph-radosgw-19.0.0-3568.g165a15e5.el 5.3 MB/s | 11 MB 00:02 2024-05-09T04:58:03.004 INFO:teuthology.orchestra.run.smithi060.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 55 kB/s | 42 kB 00:00 2024-05-09T04:58:03.006 INFO:teuthology.orchestra.run.smithi114.stdout:(19/156): librgw2-19.0.0-3568.g165a15e5.el9.x86 11 MB/s | 5.4 MB 00:00 2024-05-09T04:58:03.032 INFO:teuthology.orchestra.run.smithi114.stdout:(20/156): python3-ceph-argparse-19.0.0-3568.g16 450 kB/s | 45 kB 00:00 2024-05-09T04:58:03.057 INFO:teuthology.orchestra.run.smithi114.stdout:(21/156): python3-ceph-common-19.0.0-3568.g165a 2.6 MB/s | 135 kB 00:00 2024-05-09T04:58:03.108 INFO:teuthology.orchestra.run.smithi114.stdout:(22/156): python3-cephfs-19.0.0-3568.g165a15e5. 2.1 MB/s | 162 kB 00:00 2024-05-09T04:58:03.133 INFO:teuthology.orchestra.run.smithi114.stdout:(23/156): python3-rados-19.0.0-3568.g165a15e5.e 4.2 MB/s | 321 kB 00:00 2024-05-09T04:58:03.183 INFO:teuthology.orchestra.run.smithi114.stdout:(24/156): python3-rbd-19.0.0-3568.g165a15e5.el9 3.9 MB/s | 299 kB 00:00 2024-05-09T04:58:03.200 INFO:teuthology.orchestra.run.smithi114.stdout:(25/156): python3-rgw-19.0.0-3568.g165a15e5.el9 1.5 MB/s | 100 kB 00:00 2024-05-09T04:58:03.258 INFO:teuthology.orchestra.run.smithi114.stdout:(26/156): rbd-fuse-19.0.0-3568.g165a15e5.el9.x8 1.1 MB/s | 85 kB 00:00 2024-05-09T04:58:03.296 INFO:teuthology.orchestra.run.smithi060.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 56 kB/s | 44 kB 00:00 2024-05-09T04:58:03.376 INFO:teuthology.orchestra.run.smithi114.stdout:(27/156): rbd-nbd-19.0.0-3568.g165a15e5.el9.x86 1.4 MB/s | 171 kB 00:00 2024-05-09T04:58:03.434 INFO:teuthology.orchestra.run.smithi114.stdout:(28/156): rbd-mirror-19.0.0-3568.g165a15e5.el9. 13 MB/s | 3.1 MB 00:00 2024-05-09T04:58:03.451 INFO:teuthology.orchestra.run.smithi114.stdout:(29/156): ceph-grafana-dashboards-19.0.0-3568.g 456 kB/s | 34 kB 00:00 2024-05-09T04:58:03.484 INFO:teuthology.orchestra.run.smithi114.stdout:(30/156): ceph-mgr-cephadm-19.0.0-3568.g165a15e 2.8 MB/s | 142 kB 00:00 2024-05-09T04:58:03.655 INFO:teuthology.orchestra.run.smithi060.stdout:(94/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.6 MB/s | 16 MB 00:01 2024-05-09T04:58:03.964 INFO:teuthology.orchestra.run.smithi060.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 230 kB/s | 221 kB 00:00 2024-05-09T04:58:04.043 INFO:teuthology.orchestra.run.smithi114.stdout:(31/156): ceph-mgr-diskprediction-local-19.0.0- 13 MB/s | 7.4 MB 00:00 2024-05-09T04:58:04.081 INFO:teuthology.orchestra.run.smithi060.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 46 kB/s | 36 kB 00:00 2024-05-09T04:58:04.118 INFO:teuthology.orchestra.run.smithi114.stdout:(32/156): ceph-mgr-modules-core-19.0.0-3568.g16 3.6 MB/s | 274 kB 00:00 2024-05-09T04:58:04.177 INFO:teuthology.orchestra.run.smithi114.stdout:(33/156): ceph-mgr-dashboard-19.0.0-3568.g165a1 4.9 MB/s | 3.5 MB 00:00 2024-05-09T04:58:04.202 INFO:teuthology.orchestra.run.smithi114.stdout:(34/156): ceph-mgr-rook-19.0.0-3568.g165a15e5.e 591 kB/s | 49 kB 00:00 2024-05-09T04:58:04.253 INFO:teuthology.orchestra.run.smithi114.stdout:(35/156): ceph-prometheus-alerts-19.0.0-3568.g1 218 kB/s | 16 kB 00:00 2024-05-09T04:58:04.278 INFO:teuthology.orchestra.run.smithi114.stdout:(36/156): ceph-volume-19.0.0-3568.g165a15e5.el9 3.5 MB/s | 273 kB 00:00 2024-05-09T04:58:04.306 INFO:teuthology.orchestra.run.smithi060.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 76 kB/s | 47 kB 00:00 2024-05-09T04:58:04.345 INFO:teuthology.orchestra.run.smithi114.stdout:(37/156): cephfs-top-19.0.0-3568.g165a15e5.el9. 275 kB/s | 18 kB 00:00 2024-05-09T04:58:04.387 INFO:teuthology.orchestra.run.smithi114.stdout:(38/156): cephadm-19.0.0-3568.g165a15e5.el9.noa 5.6 MB/s | 764 kB 00:00 2024-05-09T04:58:04.470 INFO:teuthology.orchestra.run.smithi114.stdout:(39/156): fuse-2.9.9-15.el9.x86_64.rpm 641 kB/s | 80 kB 00:00 2024-05-09T04:58:04.537 INFO:teuthology.orchestra.run.smithi114.stdout:(40/156): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-05-09T04:58:04.588 INFO:teuthology.orchestra.run.smithi114.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.9 MB/s | 253 kB 00:00 2024-05-09T04:58:04.613 INFO:teuthology.orchestra.run.smithi114.stdout:(42/156): libgfortran-11.4.1-3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2024-05-09T04:58:04.664 INFO:teuthology.orchestra.run.smithi114.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-05-09T04:58:04.697 INFO:teuthology.orchestra.run.smithi114.stdout:(44/156): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.2 MB 00:00 2024-05-09T04:58:04.722 INFO:teuthology.orchestra.run.smithi114.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-05-09T04:58:04.739 INFO:teuthology.orchestra.run.smithi114.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-05-09T04:58:04.741 INFO:teuthology.orchestra.run.smithi060.stdout:(98/156): uuid-1.6.2-55.el9.x86_64.rpm 88 kB/s | 58 kB 00:00 2024-05-09T04:58:04.773 INFO:teuthology.orchestra.run.smithi114.stdout:(47/156): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-05-09T04:58:04.823 INFO:teuthology.orchestra.run.smithi060.stdout:(99/156): socat-1.7.4.1-5.el9.x86_64.rpm 355 kB/s | 305 kB 00:00 2024-05-09T04:58:04.832 INFO:teuthology.orchestra.run.smithi114.stdout:(48/156): automake-1.16.2-8.el9.noarch.rpm 7.3 MB/s | 689 kB 00:00 2024-05-09T04:58:04.865 INFO:teuthology.orchestra.run.smithi114.stdout:(49/156): bison-3.7.4-5.el9.x86_64.rpm 10 MB/s | 943 kB 00:00 2024-05-09T04:58:04.891 INFO:teuthology.orchestra.run.smithi114.stdout:(50/156): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-05-09T04:58:04.916 INFO:teuthology.orchestra.run.smithi114.stdout:(51/156): elfutils-libelf-devel-0.190-2.el9.x86 429 kB/s | 21 kB 00:00 2024-05-09T04:58:04.941 INFO:teuthology.orchestra.run.smithi114.stdout:(52/156): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-05-09T04:58:04.967 INFO:teuthology.orchestra.run.smithi114.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 650 kB/s | 33 kB 00:00 2024-05-09T04:58:05.091 INFO:teuthology.orchestra.run.smithi060.stdout:(100/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 124 kB/s | 94 kB 00:00 2024-05-09T04:58:05.165 INFO:teuthology.orchestra.run.smithi060.stdout:(101/156): lua-devel-5.4.4-4.el9.x86_64.rpm 492 kB/s | 22 kB 00:00 2024-05-09T04:58:05.290 INFO:teuthology.orchestra.run.smithi060.stdout:(102/156): protobuf-compiler-3.14.0-13.el9.x86_ 6.8 MB/s | 863 kB 00:00 2024-05-09T04:58:05.392 INFO:teuthology.orchestra.run.smithi114.stdout:(54/156): ceph-test-19.0.0-3568.g165a15e5.el9.x 11 MB/s | 49 MB 00:04 2024-05-09T04:58:05.417 INFO:teuthology.orchestra.run.smithi114.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 40 kB/s | 18 kB 00:00 2024-05-09T04:58:05.441 INFO:teuthology.orchestra.run.smithi060.stdout:(103/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 91 kB/s | 64 kB 00:00 2024-05-09T04:58:05.467 INFO:teuthology.orchestra.run.smithi114.stdout:(56/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.7 MB/s | 3.0 MB 00:00 2024-05-09T04:58:05.492 INFO:teuthology.orchestra.run.smithi114.stdout:(57/156): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 11 kB 00:00 2024-05-09T04:58:05.518 INFO:teuthology.orchestra.run.smithi114.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 904 kB/s | 45 kB 00:00 2024-05-09T04:58:05.524 INFO:teuthology.orchestra.run.smithi060.stdout:(104/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 64 kB/s | 45 kB 00:00 2024-05-09T04:58:05.543 INFO:teuthology.orchestra.run.smithi114.stdout:(59/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-05-09T04:58:05.568 INFO:teuthology.orchestra.run.smithi114.stdout:(60/156): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 582 kB 00:00 2024-05-09T04:58:05.585 INFO:teuthology.orchestra.run.smithi114.stdout:(61/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 607 kB/s | 25 kB 00:00 2024-05-09T04:58:05.610 INFO:teuthology.orchestra.run.smithi114.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-05-09T04:58:05.636 INFO:teuthology.orchestra.run.smithi114.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-05-09T04:58:05.661 INFO:teuthology.orchestra.run.smithi114.stdout:(64/156): librdkafka-1.6.1-102.el9.x86_64.rpm 3.8 MB/s | 662 kB 00:00 2024-05-09T04:58:05.717 INFO:teuthology.orchestra.run.smithi060.stdout:(105/156): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-05-09T04:58:05.728 INFO:teuthology.orchestra.run.smithi114.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-05-09T04:58:05.750 INFO:teuthology.orchestra.run.smithi060.stdout:(106/156): abseil-cpp-20211102.0-3.el9.x86_64.r 1.2 MB/s | 548 kB 00:00 2024-05-09T04:58:05.753 INFO:teuthology.orchestra.run.smithi114.stdout:(66/156): openblas-0.3.21-2.el9.x86_64.rpm 301 kB/s | 35 kB 00:00 2024-05-09T04:58:05.775 INFO:teuthology.orchestra.run.smithi060.stdout:(107/156): grpc-data-1.46.7-10.el9.noarch.rpm 78 kB/s | 19 kB 00:00 2024-05-09T04:58:05.862 INFO:teuthology.orchestra.run.smithi114.stdout:(67/156): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-05-09T04:58:05.938 INFO:teuthology.orchestra.run.smithi114.stdout:(68/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 21 MB/s | 4.4 MB 00:00 2024-05-09T04:58:05.971 INFO:teuthology.orchestra.run.smithi114.stdout:(69/156): postgresql-13.14-1.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-05-09T04:58:06.005 INFO:teuthology.orchestra.run.smithi114.stdout:(70/156): postgresql-private-libs-13.14-1.el9.x 1.9 MB/s | 133 kB 00:00 2024-05-09T04:58:06.034 INFO:teuthology.orchestra.run.smithi060.stdout:(108/156): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-05-09T04:58:06.038 INFO:teuthology.orchestra.run.smithi114.stdout:(71/156): postgresql-contrib-13.14-1.el9.x86_64 5.1 MB/s | 911 kB 00:00 2024-05-09T04:58:06.059 INFO:teuthology.orchestra.run.smithi060.stdout:(109/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 92 kB/s | 26 kB 00:00 2024-05-09T04:58:06.085 INFO:teuthology.orchestra.run.smithi060.stdout:(110/156): liboath-2.6.7-2.el9.x86_64.rpm 158 kB/s | 49 kB 00:00 2024-05-09T04:58:06.102 INFO:teuthology.orchestra.run.smithi060.stdout:(111/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-05-09T04:58:06.105 INFO:teuthology.orchestra.run.smithi114.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-05-09T04:58:06.139 INFO:teuthology.orchestra.run.smithi114.stdout:(73/156): protobuf-3.14.0-13.el9.x86_64.rpm 7.6 MB/s | 1.0 MB 00:00 2024-05-09T04:58:06.143 INFO:teuthology.orchestra.run.smithi060.stdout:(112/156): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-05-09T04:58:06.177 INFO:teuthology.orchestra.run.smithi060.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-05-09T04:58:06.202 INFO:teuthology.orchestra.run.smithi060.stdout:(114/156): python3-asyncssh-2.13.2-5.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-05-09T04:58:06.214 INFO:teuthology.orchestra.run.smithi114.stdout:(74/156): postgresql-server-13.14-1.el9.x86_64. 24 MB/s | 5.8 MB 00:00 2024-05-09T04:58:06.228 INFO:teuthology.orchestra.run.smithi060.stdout:(115/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 536 kB/s | 43 kB 00:00 2024-05-09T04:58:06.239 INFO:teuthology.orchestra.run.smithi114.stdout:(75/156): python3-devel-3.9.18-3.el9.x86_64.rpm 2.4 MB/s | 244 kB 00:00 2024-05-09T04:58:06.261 INFO:teuthology.orchestra.run.smithi060.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 552 kB/s | 32 kB 00:00 2024-05-09T04:58:06.286 INFO:teuthology.orchestra.run.smithi060.stdout:(117/156): python3-certifi-2021.10.8-2.el9.noar 180 kB/s | 15 kB 00:00 2024-05-09T04:58:06.312 INFO:teuthology.orchestra.run.smithi060.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-05-09T04:58:06.331 INFO:teuthology.orchestra.run.smithi114.stdout:(76/156): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 6.0 MB 00:00 2024-05-09T04:58:06.357 INFO:teuthology.orchestra.run.smithi114.stdout:(77/156): python3-jsonpatch-1.21-16.el9.noarch. 219 kB/s | 26 kB 00:00 2024-05-09T04:58:06.362 INFO:teuthology.orchestra.run.smithi060.stdout:(119/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.0 MB/s | 303 kB 00:00 2024-05-09T04:58:06.382 INFO:teuthology.orchestra.run.smithi114.stdout:(78/156): python3-jsonpointer-2.0-4.el9.noarch. 384 kB/s | 19 kB 00:00 2024-05-09T04:58:06.387 INFO:teuthology.orchestra.run.smithi060.stdout:(120/156): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-05-09T04:58:06.412 INFO:teuthology.orchestra.run.smithi060.stdout:(121/156): python3-colorama-0.4.4-12.el9.noarch 352 kB/s | 35 kB 00:00 2024-05-09T04:58:06.416 INFO:teuthology.orchestra.run.smithi114.stdout:(79/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-05-09T04:58:06.438 INFO:teuthology.orchestra.run.smithi060.stdout:(122/156): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-05-09T04:58:06.441 INFO:teuthology.orchestra.run.smithi114.stdout:(80/156): python3-jinja2-2.11.3-5.el9.noarch.rp 1.1 MB/s | 248 kB 00:00 2024-05-09T04:58:06.466 INFO:teuthology.orchestra.run.smithi114.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 413 kB/s | 35 kB 00:00 2024-05-09T04:58:06.500 INFO:teuthology.orchestra.run.smithi114.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-05-09T04:58:06.513 INFO:teuthology.orchestra.run.smithi060.stdout:(123/156): python3-grpcio-1.46.7-10.el9.x86_64. 20 MB/s | 2.0 MB 00:00 2024-05-09T04:58:06.525 INFO:teuthology.orchestra.run.smithi114.stdout:(83/156): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-05-09T04:58:06.538 INFO:teuthology.orchestra.run.smithi060.stdout:(124/156): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-05-09T04:58:06.550 INFO:teuthology.orchestra.run.smithi114.stdout:(84/156): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-05-09T04:58:06.555 INFO:teuthology.orchestra.run.smithi060.stdout:(125/156): python3-jaraco-8.2.1-3.el9.noarch.rp 91 kB/s | 11 kB 00:00 2024-05-09T04:58:06.575 INFO:teuthology.orchestra.run.smithi114.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 845 kB/s | 42 kB 00:00 2024-05-09T04:58:06.580 INFO:teuthology.orchestra.run.smithi060.stdout:(126/156): python3-jaraco-classes-3.2.1-5.el9.n 263 kB/s | 18 kB 00:00 2024-05-09T04:58:06.614 INFO:teuthology.orchestra.run.smithi060.stdout:(127/156): python3-jaraco-collections-3.0.0-8.e 396 kB/s | 23 kB 00:00 2024-05-09T04:58:06.631 INFO:teuthology.orchestra.run.smithi060.stdout:(128/156): python3-jaraco-functools-3.5.0-2.el9 259 kB/s | 19 kB 00:00 2024-05-09T04:58:06.651 INFO:teuthology.orchestra.run.smithi114.stdout:(86/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 5.1 MB 00:00 2024-05-09T04:58:06.656 INFO:teuthology.orchestra.run.smithi060.stdout:(129/156): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-05-09T04:58:06.676 INFO:teuthology.orchestra.run.smithi114.stdout:(87/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 159 kB 00:00 2024-05-09T04:58:06.702 INFO:teuthology.orchestra.run.smithi114.stdout:(88/156): python3-protobuf-3.14.0-13.el9.noarch 1.7 MB/s | 269 kB 00:00 2024-05-09T04:58:06.707 INFO:teuthology.orchestra.run.smithi060.stdout:(130/156): python3-kubernetes-26.1.0-2.el9.noar 14 MB/s | 1.0 MB 00:00 2024-05-09T04:58:06.724 INFO:teuthology.orchestra.run.smithi060.stdout:(131/156): python3-logutils-0.3.5-21.el9.noarch 504 kB/s | 46 kB 00:00 2024-05-09T04:58:06.727 INFO:teuthology.orchestra.run.smithi114.stdout:(89/156): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 279 kB 00:00 2024-05-09T04:58:06.749 INFO:teuthology.orchestra.run.smithi060.stdout:(132/156): python3-more-itertools-8.12.0-2.el9. 856 kB/s | 79 kB 00:00 2024-05-09T04:58:06.752 INFO:teuthology.orchestra.run.smithi114.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 674 kB/s | 51 kB 00:00 2024-05-09T04:58:06.766 INFO:teuthology.orchestra.run.smithi060.stdout:(133/156): python3-natsort-7.1.1-5.el9.noarch.r 985 kB/s | 58 kB 00:00 2024-05-09T04:58:06.799 INFO:teuthology.orchestra.run.smithi060.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 329 kB/s | 16 kB 00:00 2024-05-09T04:58:06.802 INFO:teuthology.orchestra.run.smithi114.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 534 kB/s | 54 kB 00:00 2024-05-09T04:58:06.825 INFO:teuthology.orchestra.run.smithi060.stdout:(135/156): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-05-09T04:58:06.850 INFO:teuthology.orchestra.run.smithi060.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-05-09T04:58:06.875 INFO:teuthology.orchestra.run.smithi060.stdout:(137/156): python3-pyperclip-1.8.2-2.el9.noarch 376 kB/s | 28 kB 00:00 2024-05-09T04:58:06.900 INFO:teuthology.orchestra.run.smithi060.stdout:(138/156): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-05-09T04:58:06.919 INFO:teuthology.orchestra.run.smithi114.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 249 kB/s | 42 kB 00:00 2024-05-09T04:58:06.926 INFO:teuthology.orchestra.run.smithi060.stdout:(139/156): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-05-09T04:58:06.944 INFO:teuthology.orchestra.run.smithi114.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 311 kB/s | 44 kB 00:00 2024-05-09T04:58:06.951 INFO:teuthology.orchestra.run.smithi060.stdout:(140/156): python3-rsa-4.9-2.el9.noarch.rpm 784 kB/s | 59 kB 00:00 2024-05-09T04:58:06.976 INFO:teuthology.orchestra.run.smithi060.stdout:(141/156): python3-tempora-5.0.0-2.el9.noarch.r 475 kB/s | 36 kB 00:00 2024-05-09T04:58:07.002 INFO:teuthology.orchestra.run.smithi060.stdout:(142/156): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-05-09T04:58:07.027 INFO:teuthology.orchestra.run.smithi060.stdout:(143/156): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-05-09T04:58:07.052 INFO:teuthology.orchestra.run.smithi060.stdout:(144/156): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-05-09T04:58:07.077 INFO:teuthology.orchestra.run.smithi060.stdout:(145/156): python3-werkzeug-2.0.3-3.el9.1.noarc 5.6 MB/s | 427 kB 00:00 2024-05-09T04:58:07.103 INFO:teuthology.orchestra.run.smithi060.stdout:(146/156): python3-zc-lockfile-2.0-10.el9.noarc 264 kB/s | 20 kB 00:00 2024-05-09T04:58:07.128 INFO:teuthology.orchestra.run.smithi060.stdout:(147/156): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-05-09T04:58:07.128 INFO:teuthology.orchestra.run.smithi114.stdout:(94/156): qatlib-24.02.0-1.el9.x86_64.rpm 1.0 MB/s | 221 kB 00:00 2024-05-09T04:58:07.153 INFO:teuthology.orchestra.run.smithi114.stdout:(95/156): qatlib-service-24.02.0-1.el9.x86_64.r 172 kB/s | 36 kB 00:00 2024-05-09T04:58:07.170 INFO:teuthology.orchestra.run.smithi060.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-05-09T04:58:07.337 INFO:teuthology.orchestra.run.smithi114.stdout:(96/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 227 kB/s | 47 kB 00:00 2024-05-09T04:58:07.362 INFO:teuthology.orchestra.run.smithi114.stdout:(97/156): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 305 kB 00:00 2024-05-09T04:58:07.379 INFO:teuthology.orchestra.run.smithi114.stdout:(98/156): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2024-05-09T04:58:07.429 INFO:teuthology.orchestra.run.smithi114.stdout:(99/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.4 MB/s | 94 kB 00:00 2024-05-09T04:58:07.497 INFO:teuthology.orchestra.run.smithi114.stdout:(100/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 543 kB/s | 64 kB 00:00 2024-05-09T04:58:07.514 INFO:teuthology.orchestra.run.smithi114.stdout:(101/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 534 kB/s | 45 kB 00:00 2024-05-09T04:58:07.554 INFO:teuthology.orchestra.run.smithi060.stdout:(149/156): librados2-19.0.0-3568.g165a15e5.el9. 7.5 MB/s | 3.4 MB 00:00 2024-05-09T04:58:07.956 INFO:teuthology.orchestra.run.smithi114.stdout:(102/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-05-09T04:58:08.096 INFO:teuthology.orchestra.run.smithi060.stdout:(150/156): NetworkManager-1.47.5-1.el9.x86_64.r 2.5 MB/s | 2.3 MB 00:00 2024-05-09T04:58:08.198 INFO:teuthology.orchestra.run.smithi114.stdout:(103/156): lua-devel-5.4.4-4.el9.x86_64.rpm 32 kB/s | 22 kB 00:00 2024-05-09T04:58:08.204 INFO:teuthology.orchestra.run.smithi060.stdout:(151/156): NetworkManager-team-1.47.5-1.el9.x86 386 kB/s | 42 kB 00:00 2024-05-09T04:58:08.313 INFO:teuthology.orchestra.run.smithi060.stdout:(152/156): NetworkManager-tui-1.47.5-1.el9.x86_ 2.2 MB/s | 248 kB 00:00 2024-05-09T04:58:08.521 INFO:teuthology.orchestra.run.smithi060.stdout:(153/156): openssl-3.2.1-1.el9.x86_64.rpm 6.5 MB/s | 1.3 MB 00:00 2024-05-09T04:58:08.590 INFO:teuthology.orchestra.run.smithi114.stdout:(104/156): protobuf-compiler-3.14.0-13.el9.x86_ 803 kB/s | 863 kB 00:01 2024-05-09T04:58:08.664 INFO:teuthology.orchestra.run.smithi060.stdout:(154/156): NetworkManager-libnm-1.47.5-1.el9.x8 1.6 MB/s | 1.8 MB 00:01 2024-05-09T04:58:08.747 INFO:teuthology.orchestra.run.smithi060.stdout:(155/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 11 MB/s | 2.4 MB 00:00 2024-05-09T04:58:08.881 INFO:teuthology.orchestra.run.smithi114.stdout:(105/156): grpc-data-1.46.7-10.el9.noarch.rpm 67 kB/s | 19 kB 00:00 2024-05-09T04:58:09.031 INFO:teuthology.orchestra.run.smithi060.stdout:(156/156): librbd1-19.0.0-3568.g165a15e5.el9.x8 1.7 MB/s | 3.1 MB 00:01 2024-05-09T04:58:09.035 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-05-09T04:58:09.035 INFO:teuthology.orchestra.run.smithi060.stdout:Total 8.4 MB/s | 226 MB 00:26 2024-05-09T04:58:09.215 INFO:teuthology.orchestra.run.smithi114.stdout:(106/156): gperftools-libs-2.9.1-2.el9.x86_64.r 304 kB/s | 309 kB 00:01 2024-05-09T04:58:09.349 INFO:teuthology.orchestra.run.smithi114.stdout:(107/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 195 kB/s | 26 kB 00:00 2024-05-09T04:58:09.549 INFO:teuthology.orchestra.run.smithi114.stdout:(108/156): liboath-2.6.7-2.el9.x86_64.rpm 243 kB/s | 49 kB 00:00 2024-05-09T04:58:09.666 INFO:teuthology.orchestra.run.smithi114.stdout:(109/156): abseil-cpp-20211102.0-3.el9.x86_64.r 321 kB/s | 548 kB 00:01 2024-05-09T04:58:09.749 INFO:teuthology.orchestra.run.smithi114.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 336 kB/s | 67 kB 00:00 2024-05-09T04:58:10.058 INFO:teuthology.orchestra.run.smithi114.stdout:(111/156): luarocks-3.9.2-1.el9.noarch.rpm 385 kB/s | 151 kB 00:00 2024-05-09T04:58:10.239 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-05-09T04:58:10.408 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-05-09T04:58:10.408 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-05-09T04:58:11.500 INFO:teuthology.orchestra.run.smithi114.stdout:(112/156): python3-asyncssh-2.13.2-5.el9.noarch 380 kB/s | 548 kB 00:01 2024-05-09T04:58:11.625 INFO:teuthology.orchestra.run.smithi114.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 447 kB/s | 839 kB 00:01 2024-05-09T04:58:11.742 INFO:teuthology.orchestra.run.smithi114.stdout:(114/156): python3-cachetools-4.2.4-1.el9.noarc 276 kB/s | 32 kB 00:00 2024-05-09T04:58:11.859 INFO:teuthology.orchestra.run.smithi114.stdout:(115/156): python3-certifi-2021.10.8-2.el9.noar 128 kB/s | 15 kB 00:00 2024-05-09T04:58:11.992 INFO:teuthology.orchestra.run.smithi114.stdout:(116/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 88 kB/s | 43 kB 00:00 2024-05-09T04:58:12.268 INFO:teuthology.orchestra.run.smithi114.stdout:(117/156): python3-cheroot-8.6.0-4.el9.noarch.r 420 kB/s | 172 kB 00:00 2024-05-09T04:58:12.685 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-05-09T04:58:12.686 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-05-09T04:58:12.868 INFO:teuthology.orchestra.run.smithi114.stdout:(118/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 505 kB/s | 303 kB 00:00 2024-05-09T04:58:12.985 INFO:teuthology.orchestra.run.smithi114.stdout:(119/156): python3-colorama-0.4.4-12.el9.noarch 302 kB/s | 35 kB 00:00 2024-05-09T04:58:13.132 INFO:teuthology.orchestra.run.smithi114.stdout:(120/156): python3-cherrypy-18.6.1-2.el9.noarch 315 kB/s | 358 kB 00:01 2024-05-09T04:58:13.377 INFO:teuthology.orchestra.run.smithi114.stdout:(121/156): python3-google-auth-2.29.0-1.el9.noa 524 kB/s | 206 kB 00:00 2024-05-09T04:58:13.669 INFO:teuthology.orchestra.run.smithi114.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 496 kB/s | 144 kB 00:00 2024-05-09T04:58:13.794 INFO:teuthology.orchestra.run.smithi114.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 85 kB/s | 11 kB 00:00 2024-05-09T04:58:13.911 INFO:teuthology.orchestra.run.smithi114.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 151 kB/s | 18 kB 00:00 2024-05-09T04:58:14.028 INFO:teuthology.orchestra.run.smithi114.stdout:(125/156): python3-jaraco-collections-3.0.0-8.e 198 kB/s | 23 kB 00:00 2024-05-09T04:58:14.145 INFO:teuthology.orchestra.run.smithi114.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 166 kB/s | 19 kB 00:00 2024-05-09T04:58:14.262 INFO:teuthology.orchestra.run.smithi114.stdout:(127/156): python3-jaraco-text-3.2.0-6.el9.noar 169 kB/s | 20 kB 00:00 2024-05-09T04:58:15.996 INFO:teuthology.orchestra.run.smithi114.stdout:(128/156): python3-kubernetes-26.1.0-2.el9.noar 603 kB/s | 1.0 MB 00:01 2024-05-09T04:58:16.121 INFO:teuthology.orchestra.run.smithi114.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 371 kB/s | 46 kB 00:00 2024-05-09T04:58:16.330 INFO:teuthology.orchestra.run.smithi114.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 378 kB/s | 79 kB 00:00 2024-05-09T04:58:16.455 INFO:teuthology.orchestra.run.smithi114.stdout:(131/156): python3-natsort-7.1.1-5.el9.noarch.r 462 kB/s | 58 kB 00:00 2024-05-09T04:58:16.888 INFO:teuthology.orchestra.run.smithi114.stdout:(132/156): python3-pecan-1.4.2-3.el9.noarch.rpm 628 kB/s | 272 kB 00:00 2024-05-09T04:58:17.014 INFO:teuthology.orchestra.run.smithi114.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 131 kB/s | 16 kB 00:00 2024-05-09T04:58:17.072 INFO:teuthology.orchestra.run.smithi114.stdout:(134/156): libarrow-9.0.0-11.el9.x86_64.rpm 552 kB/s | 4.4 MB 00:08 2024-05-09T04:58:17.198 INFO:teuthology.orchestra.run.smithi114.stdout:(135/156): python3-pyperclip-1.8.2-2.el9.noarch 226 kB/s | 28 kB 00:00 2024-05-09T04:58:17.223 INFO:teuthology.orchestra.run.smithi114.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 430 kB/s | 90 kB 00:00 2024-05-09T04:58:17.307 INFO:teuthology.orchestra.run.smithi114.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 283 kB/s | 31 kB 00:00 2024-05-09T04:58:17.432 INFO:teuthology.orchestra.run.smithi114.stdout:(138/156): python3-rsa-4.9-2.el9.noarch.rpm 473 kB/s | 59 kB 00:00 2024-05-09T04:58:17.549 INFO:teuthology.orchestra.run.smithi114.stdout:(139/156): python3-routes-2.5.1-5.el9.noarch.rp 579 kB/s | 188 kB 00:00 2024-05-09T04:58:17.574 INFO:teuthology.orchestra.run.smithi114.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 252 kB/s | 36 kB 00:00 2024-05-09T04:58:17.658 INFO:teuthology.orchestra.run.smithi114.stdout:(141/156): python3-typing-extensions-4.4.0-2.el 469 kB/s | 51 kB 00:00 2024-05-09T04:58:17.858 INFO:teuthology.orchestra.run.smithi114.stdout:(142/156): python3-websocket-client-1.2.3-2.el9 449 kB/s | 90 kB 00:00 2024-05-09T04:58:17.967 INFO:teuthology.orchestra.run.smithi114.stdout:(143/156): python3-webob-1.8.7-6.el9.noarch.rpm 585 kB/s | 230 kB 00:00 2024-05-09T04:58:18.008 INFO:teuthology.orchestra.run.smithi114.stdout:(144/156): python3-grpcio-1.46.7-10.el9.x86_64. 428 kB/s | 2.0 MB 00:04 2024-05-09T04:58:18.076 INFO:teuthology.orchestra.run.smithi114.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-05-09T04:58:18.301 INFO:teuthology.orchestra.run.smithi114.stdout:(146/156): re2-20211101-3.el9.x86_64.rpm 656 kB/s | 192 kB 00:00 2024-05-09T04:58:18.476 INFO:teuthology.orchestra.run.smithi114.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 691 kB/s | 427 kB 00:00 2024-05-09T04:58:18.922 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-05-09T04:58:19.013 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T04:58:19.070 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T04:58:19.139 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T04:58:19.178 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T04:58:19.203 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T04:58:19.265 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T04:58:19.460 INFO:teuthology.orchestra.run.smithi114.stdout:(148/156): librados2-19.0.0-3568.g165a15e5.el9. 2.9 MB/s | 3.4 MB 00:01 2024-05-09T04:58:19.576 INFO:teuthology.orchestra.run.smithi114.stdout:(149/156): librbd1-19.0.0-3568.g165a15e5.el9.x8 2.9 MB/s | 3.1 MB 00:01 2024-05-09T04:58:19.672 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T04:58:19.702 INFO:teuthology.orchestra.run.smithi114.stdout:(150/156): NetworkManager-1.47.5-1.el9.x86_64.r 9.6 MB/s | 2.3 MB 00:00 2024-05-09T04:58:19.730 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librados2-2:19.0.0-3568.g165a15e5.el9.x86_64 8/164 2024-05-09T04:58:19.744 INFO:teuthology.orchestra.run.smithi114.stdout:(151/156): NetworkManager-team-1.47.5-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-05-09T04:58:19.803 INFO:teuthology.orchestra.run.smithi114.stdout:(152/156): NetworkManager-libnm-1.47.5-1.el9.x8 8.1 MB/s | 1.8 MB 00:00 2024-05-09T04:58:19.828 INFO:teuthology.orchestra.run.smithi114.stdout:(153/156): NetworkManager-tui-1.47.5-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-05-09T04:58:19.838 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:19.0.0-3568.g165a15e5.el9.x86_64 8/164 2024-05-09T04:58:19.870 INFO:teuthology.orchestra.run.smithi114.stdout:(154/156): openssl-3.2.1-1.el9.x86_64.rpm 20 MB/s | 1.3 MB 00:00 2024-05-09T04:58:19.880 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs2-2:19.0.0-3568.g165a15e5.el9.x86_64 9/164 2024-05-09T04:58:19.920 INFO:teuthology.orchestra.run.smithi114.stdout:(155/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 26 MB/s | 2.4 MB 00:00 2024-05-09T04:58:19.974 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs2-2:19.0.0-3568.g165a15e5.el9.x86_64 9/164 2024-05-09T04:58:20.002 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rados-2:19.0.0-3568.g165a15e5.el9.x86_64 10/164 2024-05-09T04:58:20.189 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T04:58:20.250 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T04:58:20.277 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephsqlite-2:19.0.0-3568.g165a15e5.el9.x86_64 13/164 2024-05-09T04:58:20.374 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3568.g165a15e5.el9.x86_64 13/164 2024-05-09T04:58:20.403 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libradosstriper1-2:19.0.0-3568.g165a15e5.el9.x86 14/164 2024-05-09T04:58:20.449 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3568.g165a15e5.el9.x86 14/164 2024-05-09T04:58:20.479 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T04:58:20.505 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T04:58:20.665 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T04:58:20.803 INFO:teuthology.orchestra.run.smithi114.stdout:(156/156): thrift-0.15.0-2.el9.x86_64.rpm 594 kB/s | 1.6 MB 00:02 2024-05-09T04:58:20.807 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-05-09T04:58:20.807 INFO:teuthology.orchestra.run.smithi114.stdout:Total 10 MB/s | 226 MB 00:22 2024-05-09T04:58:20.843 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T04:58:20.993 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T04:58:21.045 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T04:58:21.163 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T04:58:21.247 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T04:58:21.451 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T04:58:21.563 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T04:58:21.592 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T04:58:21.627 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : liboath-2.6.7-2.el9.x86_64 21/164 2024-05-09T04:58:21.703 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 22/164 2024-05-09T04:58:21.751 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 23/164 2024-05-09T04:58:21.795 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 24/164 2024-05-09T04:58:21.828 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 25/164 2024-05-09T04:58:21.872 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lua-5.4.4-4.el9.x86_64 26/164 2024-05-09T04:58:21.944 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 27/164 2024-05-09T04:58:21.993 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-05-09T04:58:21.995 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 28/164 2024-05-09T04:58:22.025 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 29/164 2024-05-09T04:58:22.069 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 30/164 2024-05-09T04:58:22.143 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-05-09T04:58:22.143 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-05-09T04:58:22.147 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 31/164 2024-05-09T04:58:22.179 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-common-2:19.0.0-3568.g165a15e5.el9. 32/164 2024-05-09T04:58:22.224 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-argparse-2:19.0.0-3568.g165a15e5.el 33/164 2024-05-09T04:58:22.251 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cephfs-2:19.0.0-3568.g165a15e5.el9.x86_6 34/164 2024-05-09T04:58:22.325 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 35/164 2024-05-09T04:58:22.366 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : luarocks-3.9.2-1.el9.noarch 36/164 2024-05-09T04:58:22.445 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 37/164 2024-05-09T04:58:22.598 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 38/164 2024-05-09T04:58:22.653 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 39/164 2024-05-09T04:58:23.205 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rsa-4.9-2.el9.noarch 40/164 2024-05-09T04:58:23.293 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 41/164 2024-05-09T04:58:23.335 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 42/164 2024-05-09T04:58:23.379 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 43/164 2024-05-09T04:58:23.416 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librados-devel-2:19.0.0-3568.g165a15e5.el9.x86_6 44/164 2024-05-09T04:58:23.459 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 45/164 2024-05-09T04:58:23.621 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 46/164 2024-05-09T04:58:23.686 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 47/164 2024-05-09T04:58:23.782 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 48/164 2024-05-09T04:58:23.843 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 49/164 2024-05-09T04:58:23.918 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T04:58:23.954 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T04:58:23.991 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T04:58:24.039 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T04:58:24.074 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T04:58:24.117 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T04:58:24.159 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T04:58:24.304 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T04:58:24.309 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-05-09T04:58:24.309 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-05-09T04:58:24.347 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T04:58:24.378 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T04:58:24.419 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T04:58:24.519 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T04:58:24.553 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T04:58:24.607 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T04:58:24.635 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T04:58:24.660 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T04:58:24.747 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T04:58:24.884 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T04:58:24.918 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T04:58:24.946 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T04:58:24.980 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T04:58:25.020 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T04:58:25.060 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T04:58:25.089 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T04:58:25.124 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T04:58:25.157 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T04:58:25.199 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T04:58:25.241 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T04:58:25.332 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T04:58:25.381 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T04:58:25.411 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T04:58:25.459 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T04:58:25.570 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T04:58:25.630 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T04:58:26.394 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T04:58:26.452 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T04:58:26.485 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T04:58:26.589 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T04:58:26.645 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T04:58:26.734 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T04:58:26.771 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T04:58:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T04:58:26.993 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T04:58:27.074 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T04:58:27.771 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T04:58:27.888 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T04:58:29.222 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T04:58:29.306 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T04:58:29.335 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T04:58:29.367 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T04:58:29.698 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T04:58:29.747 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librbd1-2:19.0.0-3568.g165a15e5.el9.x86_64 101/164 2024-05-09T04:58:29.839 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:19.0.0-3568.g165a15e5.el9.x86_64 101/164 2024-05-09T04:58:29.869 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rbd-2:19.0.0-3568.g165a15e5.el9.x86_64 102/164 2024-05-09T04:58:29.905 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T04:58:30.410 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T04:58:30.433 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librgw2-2:19.0.0-3568.g165a15e5.el9.x86_64 105/164 2024-05-09T04:58:30.505 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librgw2-2:19.0.0-3568.g165a15e5.el9.x86_64 105/164 2024-05-09T04:58:30.535 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rgw-2:19.0.0-3568.g165a15e5.el9.x86_64 106/164 2024-05-09T04:58:30.797 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-05-09T04:58:30.883 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T04:58:30.924 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T04:58:30.984 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T04:58:31.025 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T04:58:31.050 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T04:58:31.108 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T04:58:31.430 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T04:58:31.526 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librados2-2:19.0.0-3568.g165a15e5.el9.x86_64 8/164 2024-05-09T04:58:31.656 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:19.0.0-3568.g165a15e5.el9.x86_64 8/164 2024-05-09T04:58:31.686 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs2-2:19.0.0-3568.g165a15e5.el9.x86_64 9/164 2024-05-09T04:58:31.777 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs2-2:19.0.0-3568.g165a15e5.el9.x86_64 9/164 2024-05-09T04:58:31.807 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rados-2:19.0.0-3568.g165a15e5.el9.x86_64 10/164 2024-05-09T04:58:31.977 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T04:58:32.037 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T04:58:32.065 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephsqlite-2:19.0.0-3568.g165a15e5.el9.x86_64 13/164 2024-05-09T04:58:32.169 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3568.g165a15e5.el9.x86_64 13/164 2024-05-09T04:58:32.191 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libradosstriper1-2:19.0.0-3568.g165a15e5.el9.x86 14/164 2024-05-09T04:58:32.254 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3568.g165a15e5.el9.x86 14/164 2024-05-09T04:58:32.275 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T04:58:32.296 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T04:58:32.485 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T04:58:32.664 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T04:58:32.789 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T04:58:32.849 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T04:58:32.926 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:19.0.0-3568.g165a15e5.el9.x86_64 107/164 2024-05-09T04:58:32.960 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T04:58:32.965 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-common-2:19.0.0-3568.g165a15e5.el9.x86_64 107/164 2024-05-09T04:58:33.025 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T04:58:33.244 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T04:58:33.348 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T04:58:33.372 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T04:58:33.406 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : liboath-2.6.7-2.el9.x86_64 21/164 2024-05-09T04:58:33.470 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 22/164 2024-05-09T04:58:33.505 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 23/164 2024-05-09T04:58:33.542 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:19.0.0-3568.g165a15e5.el9.x86_64 107/164 2024-05-09T04:58:33.549 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 24/164 2024-05-09T04:58:33.577 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-base-2:19.0.0-3568.g165a15e5.el9.x86_64 108/164 2024-05-09T04:58:33.581 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 25/164 2024-05-09T04:58:33.609 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-5.4.4-4.el9.x86_64 26/164 2024-05-09T04:58:33.652 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-base-2:19.0.0-3568.g165a15e5.el9.x86_64 108/164 2024-05-09T04:58:33.652 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T04:58:33.652 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T04:58:33.652 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:58:33.677 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 27/164 2024-05-09T04:58:33.686 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-selinux-2:19.0.0-3568.g165a15e5.el9.x86_64 109/164 2024-05-09T04:58:33.739 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 28/164 2024-05-09T04:58:33.770 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 29/164 2024-05-09T04:58:33.806 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 30/164 2024-05-09T04:58:33.882 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 31/164 2024-05-09T04:58:33.916 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-common-2:19.0.0-3568.g165a15e5.el9. 32/164 2024-05-09T04:58:33.960 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-argparse-2:19.0.0-3568.g165a15e5.el 33/164 2024-05-09T04:58:33.988 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cephfs-2:19.0.0-3568.g165a15e5.el9.x86_6 34/164 2024-05-09T04:58:34.057 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 35/164 2024-05-09T04:58:34.103 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : luarocks-3.9.2-1.el9.noarch 36/164 2024-05-09T04:58:34.177 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 37/164 2024-05-09T04:58:34.317 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 38/164 2024-05-09T04:58:34.372 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 39/164 2024-05-09T04:58:34.890 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rsa-4.9-2.el9.noarch 40/164 2024-05-09T04:58:34.975 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 41/164 2024-05-09T04:58:35.030 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 42/164 2024-05-09T04:58:35.073 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 43/164 2024-05-09T04:58:35.110 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librados-devel-2:19.0.0-3568.g165a15e5.el9.x86_6 44/164 2024-05-09T04:58:35.153 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 45/164 2024-05-09T04:58:35.300 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 46/164 2024-05-09T04:58:35.338 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 47/164 2024-05-09T04:58:35.429 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 48/164 2024-05-09T04:58:35.503 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 49/164 2024-05-09T04:58:35.574 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T04:58:35.606 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T04:58:35.643 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T04:58:35.689 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T04:58:35.725 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T04:58:35.768 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T04:58:35.810 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T04:58:35.953 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T04:58:35.999 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T04:58:36.030 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T04:58:36.071 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T04:58:36.164 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T04:58:36.196 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T04:58:36.237 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T04:58:36.270 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T04:58:36.295 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T04:58:36.392 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T04:58:36.521 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T04:58:36.560 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T04:58:36.588 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T04:58:36.623 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T04:58:36.661 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T04:58:36.694 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T04:58:36.723 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T04:58:36.758 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T04:58:36.799 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T04:58:36.832 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T04:58:36.867 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T04:58:36.953 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T04:58:36.997 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T04:58:37.028 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T04:58:37.075 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T04:58:37.173 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T04:58:37.229 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T04:58:37.950 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T04:58:38.002 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T04:58:38.035 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T04:58:38.133 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T04:58:38.185 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T04:58:38.280 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T04:58:38.320 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T04:58:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T04:58:38.535 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T04:58:38.614 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T04:58:39.278 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T04:58:39.382 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T04:58:40.633 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T04:58:40.705 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T04:58:40.733 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T04:58:40.766 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T04:58:41.080 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T04:58:41.121 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librbd1-2:19.0.0-3568.g165a15e5.el9.x86_64 101/164 2024-05-09T04:58:41.212 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:19.0.0-3568.g165a15e5.el9.x86_64 101/164 2024-05-09T04:58:41.242 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rbd-2:19.0.0-3568.g165a15e5.el9.x86_64 102/164 2024-05-09T04:58:41.279 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T04:58:41.762 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T04:58:41.799 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librgw2-2:19.0.0-3568.g165a15e5.el9.x86_64 105/164 2024-05-09T04:58:41.870 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librgw2-2:19.0.0-3568.g165a15e5.el9.x86_64 105/164 2024-05-09T04:58:41.900 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rgw-2:19.0.0-3568.g165a15e5.el9.x86_64 106/164 2024-05-09T04:58:44.186 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-3568.g165a15e5.el9.x86_64 107/164 2024-05-09T04:58:44.221 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-common-2:19.0.0-3568.g165a15e5.el9.x86_64 107/164 2024-05-09T04:58:44.764 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-3568.g165a15e5.el9.x86_64 107/164 2024-05-09T04:58:44.800 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-base-2:19.0.0-3568.g165a15e5.el9.x86_64 108/164 2024-05-09T04:58:44.882 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-base-2:19.0.0-3568.g165a15e5.el9.x86_64 108/164 2024-05-09T04:58:44.882 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T04:58:44.882 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T04:58:44.882 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:58:44.917 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-selinux-2:19.0.0-3568.g165a15e5.el9.x86_64 109/164 2024-05-09T04:58:45.624 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3568.g165a15e5.el9.x86_64 109/164 2024-05-09T04:58:45.624 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /sys 2024-05-09T04:58:45.624 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /proc 2024-05-09T04:58:45.624 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /mnt 2024-05-09T04:58:45.625 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /var/tmp 2024-05-09T04:58:45.625 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /home 2024-05-09T04:58:45.625 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /root 2024-05-09T04:58:45.625 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /tmp 2024-05-09T04:58:45.625 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:58:45.820 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-volume-2:19.0.0-3568.g165a15e5.el9.noarch 110/164 2024-05-09T04:58:45.837 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-volume-2:19.0.0-3568.g165a15e5.el9.noarch 110/164 2024-05-09T04:58:45.837 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T04:58:45.837 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T04:58:45.837 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:58:47.390 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-osd-2:19.0.0-3568.g165a15e5.el9.x86_64 111/164 2024-05-09T04:58:47.417 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-osd-2:19.0.0-3568.g165a15e5.el9.x86_64 111/164 2024-05-09T04:58:47.418 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T04:58:47.418 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T04:58:47.418 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T04:58:47.418 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T04:58:47.418 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:58:47.666 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mds-2:19.0.0-3568.g165a15e5.el9.x86_64 112/164 2024-05-09T04:58:47.690 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mds-2:19.0.0-3568.g165a15e5.el9.x86_64 112/164 2024-05-09T04:58:47.690 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T04:58:47.690 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T04:58:47.690 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T04:58:47.690 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T04:58:47.690 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:58:48.158 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mon-2:19.0.0-3568.g165a15e5.el9.x86_64 113/164 2024-05-09T04:58:48.182 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mon-2:19.0.0-3568.g165a15e5.el9.x86_64 113/164 2024-05-09T04:58:48.183 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T04:58:48.183 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T04:58:48.183 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T04:58:48.183 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T04:58:48.183 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:58:48.297 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T04:58:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T04:58:48.419 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T04:58:48.623 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T04:58:48.676 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T04:58:48.756 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T04:58:48.846 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T04:58:49.015 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T04:58:49.088 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T04:58:49.125 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T04:58:49.152 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T04:58:49.615 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:19.0.0-3568.g165a15e5.el9.noarch 125/164 2024-05-09T04:58:49.642 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephadm-2:19.0.0-3568.g165a15e5.el9.noarch 125/164 2024-05-09T04:58:49.675 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3568.g165a15e5.e 126/164 2024-05-09T04:58:49.738 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3568.g165a15e5. 127/164 2024-05-09T04:58:49.765 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3568.g165a15e5.el9.noa 128/164 2024-05-09T04:58:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3568.g165a15e5.el9.noa 128/164 2024-05-09T04:58:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3568.g165a15e5.el9.n 129/164 2024-05-09T04:58:51.905 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3568.g165a15e5.el9.n 129/164 2024-05-09T04:58:51.934 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3568.g165 130/164 2024-05-09T04:58:52.053 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3568.g165 130/164 2024-05-09T04:58:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3568.g165a15e5.el 131/164 2024-05-09T04:58:52.237 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-2:19.0.0-3568.g165a15e5.el9.x86_64 132/164 2024-05-09T04:58:52.262 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3568.g165a15e5.el9.x86_64 132/164 2024-05-09T04:58:52.262 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T04:58:52.262 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T04:58:52.262 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T04:58:52.262 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T04:58:52.262 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:58:52.312 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-rook-2:19.0.0-3568.g165a15e5.el9.noarch 133/164 2024-05-09T04:58:52.324 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3568.g165a15e5.el9.noarch 133/164 2024-05-09T04:58:52.429 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-2:19.0.0-3568.g165a15e5.el9.x86_64 134/164 2024-05-09T04:58:52.486 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-fuse-2:19.0.0-3568.g165a15e5.el9.x86_64 135/164 2024-05-09T04:58:53.461 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T04:58:53.489 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-radosgw-2:19.0.0-3568.g165a15e5.el9.x86_64 137/164 2024-05-09T04:58:53.513 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3568.g165a15e5.el9.x86_64 137/164 2024-05-09T04:58:53.514 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T04:58:53.514 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T04:58:53.514 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T04:58:53.514 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T04:58:53.514 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:58:53.557 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3568.g165a1 138/164 2024-05-09T04:58:53.581 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3568.g165a1 138/164 2024-05-09T04:58:53.581 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T04:58:53.581 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T04:58:53.581 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:58:53.632 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephfs-mirror-2:19.0.0-3568.g165a15e5.el9.x86_64 139/164 2024-05-09T04:58:53.656 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3568.g165a15e5.el9.x86_64 139/164 2024-05-09T04:58:53.657 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T04:58:53.657 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T04:58:53.657 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T04:58:53.657 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T04:58:53.657 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:58:53.975 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-mirror-2:19.0.0-3568.g165a15e5.el9.x86_64 140/164 2024-05-09T04:58:53.999 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3568.g165a15e5.el9.x86_64 140/164 2024-05-09T04:58:53.999 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T04:58:53.999 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T04:58:54.000 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T04:58:54.000 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T04:58:54.000 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:58:56.503 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3568.g165a15e5.el9.x86_64 109/164 2024-05-09T04:58:56.503 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /sys 2024-05-09T04:58:56.503 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /proc 2024-05-09T04:58:56.503 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /mnt 2024-05-09T04:58:56.503 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /var/tmp 2024-05-09T04:58:56.503 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /home 2024-05-09T04:58:56.503 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /root 2024-05-09T04:58:56.503 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /tmp 2024-05-09T04:58:56.504 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:58:56.672 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-volume-2:19.0.0-3568.g165a15e5.el9.noarch 110/164 2024-05-09T04:58:56.688 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-volume-2:19.0.0-3568.g165a15e5.el9.noarch 110/164 2024-05-09T04:58:56.688 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T04:58:56.688 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T04:58:56.688 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:58:58.167 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-osd-2:19.0.0-3568.g165a15e5.el9.x86_64 111/164 2024-05-09T04:58:58.193 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-osd-2:19.0.0-3568.g165a15e5.el9.x86_64 111/164 2024-05-09T04:58:58.193 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T04:58:58.193 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T04:58:58.194 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T04:58:58.194 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T04:58:58.194 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:58:58.434 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mds-2:19.0.0-3568.g165a15e5.el9.x86_64 112/164 2024-05-09T04:58:58.457 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mds-2:19.0.0-3568.g165a15e5.el9.x86_64 112/164 2024-05-09T04:58:58.457 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T04:58:58.457 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T04:58:58.457 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T04:58:58.457 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T04:58:58.458 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:58:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-test-2:19.0.0-3568.g165a15e5.el9.x86_64 141/164 2024-05-09T04:58:58.893 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mon-2:19.0.0-3568.g165a15e5.el9.x86_64 113/164 2024-05-09T04:58:58.916 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mon-2:19.0.0-3568.g165a15e5.el9.x86_64 113/164 2024-05-09T04:58:58.916 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T04:58:58.916 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T04:58:58.917 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T04:58:58.917 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T04:58:58.917 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:58:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-fuse-2:19.0.0-3568.g165a15e5.el9.x86_64 142/164 2024-05-09T04:58:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-nbd-2:19.0.0-3568.g165a15e5.el9.x86_64 143/164 2024-05-09T04:58:59.001 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T04:58:59.014 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T04:58:59.028 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T04:58:59.063 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T04:58:59.118 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T04:58:59.233 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephfs-shell-2:19.0.0-3568.g165a15e5.el9.x86_64 146/164 2024-05-09T04:58:59.284 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T04:58:59.311 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T04:58:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs-devel-2:19.0.0-3568.g165a15e5.el9.x86_ 148/164 2024-05-09T04:58:59.343 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T04:58:59.360 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T04:58:59.389 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T04:58:59.428 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T04:58:59.517 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T04:58:59.664 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T04:58:59.702 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T04:58:59.742 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T04:58:59.770 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T04:59:00.040 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T04:59:00.087 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T04:59:00.108 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T04:59:00.241 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.0.0-3568.g165a15e5.el9.noarch 125/164 2024-05-09T04:59:00.270 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephadm-2:19.0.0-3568.g165a15e5.el9.noarch 125/164 2024-05-09T04:59:00.302 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3568.g165a15e5.e 126/164 2024-05-09T04:59:00.364 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3568.g165a15e5. 127/164 2024-05-09T04:59:00.393 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3568.g165a15e5.el9.noa 128/164 2024-05-09T04:59:01.016 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephfs-top-2:19.0.0-3568.g165a15e5.el9.noarch 152/164 2024-05-09T04:59:01.108 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T04:59:01.162 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T04:59:01.263 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T04:59:01.295 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T04:59:01.321 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T04:59:01.321 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T04:59:01.405 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T04:59:01.422 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3568.g165a15e5.el9.noa 128/164 2024-05-09T04:59:01.438 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T04:59:01.438 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T04:59:01.454 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3568.g165a15e5.el9.n 129/164 2024-05-09T04:59:01.523 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T04:59:01.548 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T04:59:01.553 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T04:59:01.553 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T04:59:01.741 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T04:59:01.741 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T04:59:01.835 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T04:59:01.910 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T04:59:02.411 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3568.g165a15e5.el9.n 129/164 2024-05-09T04:59:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3568.g165 130/164 2024-05-09T04:59:02.548 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3568.g165 130/164 2024-05-09T04:59:02.708 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3568.g165a15e5.el 131/164 2024-05-09T04:59:02.740 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-2:19.0.0-3568.g165a15e5.el9.x86_64 132/164 2024-05-09T04:59:02.764 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3568.g165a15e5.el9.x86_64 132/164 2024-05-09T04:59:02.764 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T04:59:02.764 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T04:59:02.764 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T04:59:02.764 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T04:59:02.765 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:02.815 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-rook-2:19.0.0-3568.g165a15e5.el9.noarch 133/164 2024-05-09T04:59:02.827 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3568.g165a15e5.el9.noarch 133/164 2024-05-09T04:59:02.932 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-2:19.0.0-3568.g165a15e5.el9.x86_64 134/164 2024-05-09T04:59:02.989 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-fuse-2:19.0.0-3568.g165a15e5.el9.x86_64 135/164 2024-05-09T04:59:03.914 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T04:59:03.943 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-radosgw-2:19.0.0-3568.g165a15e5.el9.x86_64 137/164 2024-05-09T04:59:03.967 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3568.g165a15e5.el9.x86_64 137/164 2024-05-09T04:59:03.967 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T04:59:03.967 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T04:59:03.967 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T04:59:03.967 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T04:59:03.967 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3568.g165a1 138/164 2024-05-09T04:59:04.033 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3568.g165a1 138/164 2024-05-09T04:59:04.033 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T04:59:04.033 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T04:59:04.033 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:04.086 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-mirror-2:19.0.0-3568.g165a15e5.el9.x86_64 139/164 2024-05-09T04:59:04.109 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3568.g165a15e5.el9.x86_64 139/164 2024-05-09T04:59:04.109 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T04:59:04.109 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T04:59:04.109 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T04:59:04.109 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T04:59:04.110 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:04.412 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-mirror-2:19.0.0-3568.g165a15e5.el9.x86_64 140/164 2024-05-09T04:59:04.435 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3568.g165a15e5.el9.x86_64 140/164 2024-05-09T04:59:04.435 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T04:59:04.435 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T04:59:04.435 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T04:59:04.435 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T04:59:04.435 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:07.557 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T04:59:07.557 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-2:19.0.0-3568.g165a15e5.el9.x86_64 1/164 2024-05-09T04:59:07.557 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-base-2:19.0.0-3568.g165a15e5.el9.x86_64 2/164 2024-05-09T04:59:07.558 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-common-2:19.0.0-3568.g165a15e5.el9.x86_64 3/164 2024-05-09T04:59:07.558 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-fuse-2:19.0.0-3568.g165a15e5.el9.x86_64 4/164 2024-05-09T04:59:07.558 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3568.g165a1 5/164 2024-05-09T04:59:07.558 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mds-2:19.0.0-3568.g165a15e5.el9.x86_64 6/164 2024-05-09T04:59:07.558 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-2:19.0.0-3568.g165a15e5.el9.x86_64 7/164 2024-05-09T04:59:07.558 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mon-2:19.0.0-3568.g165a15e5.el9.x86_64 8/164 2024-05-09T04:59:07.558 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-osd-2:19.0.0-3568.g165a15e5.el9.x86_64 9/164 2024-05-09T04:59:07.558 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-radosgw-2:19.0.0-3568.g165a15e5.el9.x86_64 10/164 2024-05-09T04:59:07.558 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-selinux-2:19.0.0-3568.g165a15e5.el9.x86_64 11/164 2024-05-09T04:59:07.558 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-test-2:19.0.0-3568.g165a15e5.el9.x86_64 12/164 2024-05-09T04:59:07.558 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephfs-mirror-2:19.0.0-3568.g165a15e5.el9.x86_64 13/164 2024-05-09T04:59:07.558 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephfs-shell-2:19.0.0-3568.g165a15e5.el9.x86_64 14/164 2024-05-09T04:59:07.558 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs-devel-2:19.0.0-3568.g165a15e5.el9.x86_ 15/164 2024-05-09T04:59:07.559 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs2-2:19.0.0-3568.g165a15e5.el9.x86_64 16/164 2024-05-09T04:59:07.559 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephsqlite-2:19.0.0-3568.g165a15e5.el9.x86_64 17/164 2024-05-09T04:59:07.559 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados-devel-2:19.0.0-3568.g165a15e5.el9.x86_6 18/164 2024-05-09T04:59:07.559 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libradosstriper1-2:19.0.0-3568.g165a15e5.el9.x86 19/164 2024-05-09T04:59:07.559 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librgw2-2:19.0.0-3568.g165a15e5.el9.x86_64 20/164 2024-05-09T04:59:07.559 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3568.g165a15e5.el 21/164 2024-05-09T04:59:07.559 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-common-2:19.0.0-3568.g165a15e5.el9. 22/164 2024-05-09T04:59:07.559 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cephfs-2:19.0.0-3568.g165a15e5.el9.x86_6 23/164 2024-05-09T04:59:07.559 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rados-2:19.0.0-3568.g165a15e5.el9.x86_64 24/164 2024-05-09T04:59:07.559 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rbd-2:19.0.0-3568.g165a15e5.el9.x86_64 25/164 2024-05-09T04:59:07.559 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rgw-2:19.0.0-3568.g165a15e5.el9.x86_64 26/164 2024-05-09T04:59:07.559 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-fuse-2:19.0.0-3568.g165a15e5.el9.x86_64 27/164 2024-05-09T04:59:07.560 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-mirror-2:19.0.0-3568.g165a15e5.el9.x86_64 28/164 2024-05-09T04:59:07.560 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-nbd-2:19.0.0-3568.g165a15e5.el9.x86_64 29/164 2024-05-09T04:59:07.571 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3568.g165a15e5. 30/164 2024-05-09T04:59:07.571 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3568.g165a15e5.el9.noa 31/164 2024-05-09T04:59:07.572 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3568.g165a15e5.el9.n 32/164 2024-05-09T04:59:07.572 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3568.g165 33/164 2024-05-09T04:59:07.572 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3568.g165a15e5.el 34/164 2024-05-09T04:59:07.572 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3568.g165a15e5.el9.noarch 35/164 2024-05-09T04:59:07.572 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3568.g165a15e5.e 36/164 2024-05-09T04:59:07.572 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-volume-2:19.0.0-3568.g165a15e5.el9.noarch 37/164 2024-05-09T04:59:07.572 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephadm-2:19.0.0-3568.g165a15e5.el9.noarch 38/164 2024-05-09T04:59:07.572 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephfs-top-2:19.0.0-3568.g165a15e5.el9.noarch 39/164 2024-05-09T04:59:07.572 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T04:59:07.572 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T04:59:07.572 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T04:59:07.572 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T04:59:07.572 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T04:59:07.572 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T04:59:07.573 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T04:59:07.573 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T04:59:07.573 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T04:59:07.573 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T04:59:07.573 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T04:59:07.573 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T04:59:07.573 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T04:59:07.573 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T04:59:07.573 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T04:59:07.573 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T04:59:07.573 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T04:59:07.573 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T04:59:07.573 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T04:59:07.573 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T04:59:07.574 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T04:59:07.574 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T04:59:07.574 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T04:59:07.574 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T04:59:07.574 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T04:59:07.574 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T04:59:07.574 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T04:59:07.574 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T04:59:07.574 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T04:59:07.574 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T04:59:07.574 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T04:59:07.574 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T04:59:07.574 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T04:59:07.575 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T04:59:07.575 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T04:59:07.575 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T04:59:07.575 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T04:59:07.575 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T04:59:07.575 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T04:59:07.575 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T04:59:07.575 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T04:59:07.575 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T04:59:07.575 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T04:59:07.575 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T04:59:07.575 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T04:59:07.575 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T04:59:07.575 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T04:59:07.576 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T04:59:07.576 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T04:59:07.576 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T04:59:07.576 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T04:59:07.576 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T04:59:07.576 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T04:59:07.576 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T04:59:07.576 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T04:59:07.576 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T04:59:07.576 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T04:59:07.576 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T04:59:07.576 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T04:59:07.576 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T04:59:07.576 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T04:59:07.576 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T04:59:07.577 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T04:59:07.577 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T04:59:07.577 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T04:59:07.577 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T04:59:07.577 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T04:59:07.577 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T04:59:07.577 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T04:59:07.577 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T04:59:07.577 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T04:59:07.577 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T04:59:07.578 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T04:59:07.578 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T04:59:07.578 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T04:59:07.578 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T04:59:07.578 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T04:59:07.578 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T04:59:07.579 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T04:59:07.579 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T04:59:07.579 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T04:59:07.579 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T04:59:07.579 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T04:59:07.579 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T04:59:07.579 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T04:59:07.579 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T04:59:07.580 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T04:59:07.580 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T04:59:07.580 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T04:59:07.580 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T04:59:07.580 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T04:59:07.580 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T04:59:07.580 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T04:59:07.581 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T04:59:07.581 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T04:59:07.581 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T04:59:07.581 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T04:59:07.581 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T04:59:07.581 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T04:59:07.581 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T04:59:07.581 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T04:59:07.581 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T04:59:07.581 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T04:59:07.582 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T04:59:07.582 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T04:59:07.582 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T04:59:07.582 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T04:59:07.582 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T04:59:07.582 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T04:59:07.582 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:19.0.0-3568.g165a15e5.el9.x86_64 149/164 2024-05-09T04:59:07.582 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T04:59:07.582 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:19.0.0-3568.g165a15e5.el9.x86_64 151/164 2024-05-09T04:59:07.582 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T04:59:07.582 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T04:59:07.582 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T04:59:07.582 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T04:59:07.582 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T04:59:07.583 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T04:59:07.583 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T04:59:07.583 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T04:59:07.583 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T04:59:07.583 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T04:59:07.583 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T04:59:07.583 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T04:59:09.024 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-test-2:19.0.0-3568.g165a15e5.el9.x86_64 141/164 2024-05-09T04:59:09.065 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-fuse-2:19.0.0-3568.g165a15e5.el9.x86_64 142/164 2024-05-09T04:59:09.106 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-nbd-2:19.0.0-3568.g165a15e5.el9.x86_64 143/164 2024-05-09T04:59:09.138 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T04:59:09.165 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T04:59:09.354 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-shell-2:19.0.0-3568.g165a15e5.el9.x86_64 146/164 2024-05-09T04:59:09.403 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T04:59:09.434 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs-devel-2:19.0.0-3568.g165a15e5.el9.x86_ 148/164 2024-05-09T04:59:09.463 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T04:59:09.510 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T04:59:10.095 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T04:59:10.132 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T04:59:10.153 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T04:59:10.378 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T04:59:10.378 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:10.378 INFO:teuthology.orchestra.run.smithi060.stdout:Upgraded: 2024-05-09T04:59:10.378 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T04:59:10.378 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T04:59:10.378 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T04:59:10.379 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T04:59:10.379 INFO:teuthology.orchestra.run.smithi060.stdout: librados2-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.379 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.379 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T04:59:10.379 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T04:59:10.379 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-05-09T04:59:10.379 INFO:teuthology.orchestra.run.smithi060.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T04:59:10.379 INFO:teuthology.orchestra.run.smithi060.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T04:59:10.379 INFO:teuthology.orchestra.run.smithi060.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T04:59:10.379 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T04:59:10.379 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.379 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.379 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.379 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.380 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:10.380 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.380 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.380 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.380 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:10.380 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:10.380 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:10.380 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:10.380 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:10.380 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.380 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.380 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:10.381 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.381 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.381 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.381 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:10.381 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:10.381 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-mirror-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.381 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-shell-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.381 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-top-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:10.381 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T04:59:10.381 INFO:teuthology.orchestra.run.smithi060.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T04:59:10.381 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T04:59:10.381 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T04:59:10.382 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T04:59:10.382 INFO:teuthology.orchestra.run.smithi060.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T04:59:10.382 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T04:59:10.382 INFO:teuthology.orchestra.run.smithi060.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T04:59:10.382 INFO:teuthology.orchestra.run.smithi060.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T04:59:10.382 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T04:59:10.382 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T04:59:10.382 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.382 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.382 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.382 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T04:59:10.382 INFO:teuthology.orchestra.run.smithi060.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T04:59:10.382 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T04:59:10.383 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T04:59:10.383 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T04:59:10.383 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.383 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.383 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T04:59:10.383 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.383 INFO:teuthology.orchestra.run.smithi060.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T04:59:10.383 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T04:59:10.383 INFO:teuthology.orchestra.run.smithi060.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T04:59:10.383 INFO:teuthology.orchestra.run.smithi060.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T04:59:10.383 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T04:59:10.383 INFO:teuthology.orchestra.run.smithi060.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T04:59:10.383 INFO:teuthology.orchestra.run.smithi060.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T04:59:10.383 INFO:teuthology.orchestra.run.smithi060.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T04:59:10.384 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T04:59:10.384 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T04:59:10.384 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T04:59:10.384 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T04:59:10.384 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T04:59:10.384 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T04:59:10.384 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T04:59:10.384 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T04:59:10.384 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T04:59:10.384 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T04:59:10.384 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T04:59:10.384 INFO:teuthology.orchestra.run.smithi060.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T04:59:10.385 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T04:59:10.385 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T04:59:10.385 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T04:59:10.385 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.385 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.385 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.385 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T04:59:10.385 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T04:59:10.385 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T04:59:10.385 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T04:59:10.385 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T04:59:10.385 INFO:teuthology.orchestra.run.smithi060.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T04:59:10.385 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T04:59:10.385 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T04:59:10.386 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T04:59:10.386 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T04:59:10.386 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T04:59:10.386 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T04:59:10.386 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T04:59:10.386 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T04:59:10.386 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T04:59:10.387 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T04:59:10.387 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T04:59:10.387 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T04:59:10.387 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T04:59:10.387 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T04:59:10.387 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T04:59:10.387 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T04:59:10.387 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T04:59:10.387 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T04:59:10.387 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T04:59:10.387 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T04:59:10.388 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T04:59:10.388 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T04:59:10.388 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T04:59:10.388 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T04:59:10.388 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T04:59:10.388 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T04:59:10.388 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T04:59:10.388 INFO:teuthology.orchestra.run.smithi060.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T04:59:10.388 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T04:59:10.388 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T04:59:10.388 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T04:59:10.388 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T04:59:10.388 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T04:59:10.388 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T04:59:10.389 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.389 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.389 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T04:59:10.389 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T04:59:10.389 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T04:59:10.389 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.389 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T04:59:10.389 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T04:59:10.389 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T04:59:10.389 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T04:59:10.389 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T04:59:10.389 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T04:59:10.389 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T04:59:10.389 INFO:teuthology.orchestra.run.smithi060.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T04:59:10.390 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T04:59:10.390 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T04:59:10.390 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T04:59:10.390 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T04:59:10.390 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T04:59:10.390 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T04:59:10.390 INFO:teuthology.orchestra.run.smithi060.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T04:59:10.390 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.390 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.390 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:10.390 INFO:teuthology.orchestra.run.smithi060.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T04:59:10.390 INFO:teuthology.orchestra.run.smithi060.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T04:59:10.390 INFO:teuthology.orchestra.run.smithi060.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T04:59:10.391 INFO:teuthology.orchestra.run.smithi060.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T04:59:10.391 INFO:teuthology.orchestra.run.smithi060.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T04:59:10.391 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T04:59:10.391 INFO:teuthology.orchestra.run.smithi060.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T04:59:10.391 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:10.391 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-05-09T04:59:10.805 DEBUG:teuthology.parallel:result is None 2024-05-09T04:59:10.973 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-top-2:19.0.0-3568.g165a15e5.el9.noarch 152/164 2024-05-09T04:59:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T04:59:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T04:59:11.199 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T04:59:11.232 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T04:59:11.258 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T04:59:11.258 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T04:59:11.343 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T04:59:11.375 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T04:59:11.376 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T04:59:11.460 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T04:59:11.485 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T04:59:11.491 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T04:59:11.492 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T04:59:11.687 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T04:59:11.687 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T04:59:11.772 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T04:59:11.845 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T04:59:17.363 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T04:59:17.363 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-2:19.0.0-3568.g165a15e5.el9.x86_64 1/164 2024-05-09T04:59:17.363 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-base-2:19.0.0-3568.g165a15e5.el9.x86_64 2/164 2024-05-09T04:59:17.363 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-common-2:19.0.0-3568.g165a15e5.el9.x86_64 3/164 2024-05-09T04:59:17.363 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-fuse-2:19.0.0-3568.g165a15e5.el9.x86_64 4/164 2024-05-09T04:59:17.363 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3568.g165a1 5/164 2024-05-09T04:59:17.363 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mds-2:19.0.0-3568.g165a15e5.el9.x86_64 6/164 2024-05-09T04:59:17.364 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-2:19.0.0-3568.g165a15e5.el9.x86_64 7/164 2024-05-09T04:59:17.364 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mon-2:19.0.0-3568.g165a15e5.el9.x86_64 8/164 2024-05-09T04:59:17.364 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-osd-2:19.0.0-3568.g165a15e5.el9.x86_64 9/164 2024-05-09T04:59:17.364 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-radosgw-2:19.0.0-3568.g165a15e5.el9.x86_64 10/164 2024-05-09T04:59:17.364 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-selinux-2:19.0.0-3568.g165a15e5.el9.x86_64 11/164 2024-05-09T04:59:17.364 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-test-2:19.0.0-3568.g165a15e5.el9.x86_64 12/164 2024-05-09T04:59:17.364 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-mirror-2:19.0.0-3568.g165a15e5.el9.x86_64 13/164 2024-05-09T04:59:17.364 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-shell-2:19.0.0-3568.g165a15e5.el9.x86_64 14/164 2024-05-09T04:59:17.364 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs-devel-2:19.0.0-3568.g165a15e5.el9.x86_ 15/164 2024-05-09T04:59:17.364 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs2-2:19.0.0-3568.g165a15e5.el9.x86_64 16/164 2024-05-09T04:59:17.364 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephsqlite-2:19.0.0-3568.g165a15e5.el9.x86_64 17/164 2024-05-09T04:59:17.364 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados-devel-2:19.0.0-3568.g165a15e5.el9.x86_6 18/164 2024-05-09T04:59:17.365 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libradosstriper1-2:19.0.0-3568.g165a15e5.el9.x86 19/164 2024-05-09T04:59:17.365 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librgw2-2:19.0.0-3568.g165a15e5.el9.x86_64 20/164 2024-05-09T04:59:17.365 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3568.g165a15e5.el 21/164 2024-05-09T04:59:17.365 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-common-2:19.0.0-3568.g165a15e5.el9. 22/164 2024-05-09T04:59:17.365 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cephfs-2:19.0.0-3568.g165a15e5.el9.x86_6 23/164 2024-05-09T04:59:17.365 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rados-2:19.0.0-3568.g165a15e5.el9.x86_64 24/164 2024-05-09T04:59:17.365 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rbd-2:19.0.0-3568.g165a15e5.el9.x86_64 25/164 2024-05-09T04:59:17.365 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rgw-2:19.0.0-3568.g165a15e5.el9.x86_64 26/164 2024-05-09T04:59:17.365 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-fuse-2:19.0.0-3568.g165a15e5.el9.x86_64 27/164 2024-05-09T04:59:17.365 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-mirror-2:19.0.0-3568.g165a15e5.el9.x86_64 28/164 2024-05-09T04:59:17.365 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-nbd-2:19.0.0-3568.g165a15e5.el9.x86_64 29/164 2024-05-09T04:59:17.377 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3568.g165a15e5. 30/164 2024-05-09T04:59:17.378 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3568.g165a15e5.el9.noa 31/164 2024-05-09T04:59:17.378 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3568.g165a15e5.el9.n 32/164 2024-05-09T04:59:17.378 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3568.g165 33/164 2024-05-09T04:59:17.378 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3568.g165a15e5.el 34/164 2024-05-09T04:59:17.378 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3568.g165a15e5.el9.noarch 35/164 2024-05-09T04:59:17.378 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3568.g165a15e5.e 36/164 2024-05-09T04:59:17.378 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-volume-2:19.0.0-3568.g165a15e5.el9.noarch 37/164 2024-05-09T04:59:17.378 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephadm-2:19.0.0-3568.g165a15e5.el9.noarch 38/164 2024-05-09T04:59:17.378 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-top-2:19.0.0-3568.g165a15e5.el9.noarch 39/164 2024-05-09T04:59:17.378 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T04:59:17.378 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T04:59:17.378 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T04:59:17.379 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T04:59:17.379 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T04:59:17.379 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T04:59:17.379 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T04:59:17.379 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T04:59:17.379 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T04:59:17.379 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T04:59:17.379 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T04:59:17.379 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T04:59:17.379 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T04:59:17.379 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T04:59:17.379 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T04:59:17.379 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T04:59:17.380 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T04:59:17.380 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T04:59:17.380 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T04:59:17.380 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T04:59:17.380 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T04:59:17.380 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T04:59:17.380 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T04:59:17.380 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T04:59:17.380 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T04:59:17.380 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T04:59:17.380 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T04:59:17.380 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T04:59:17.381 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T04:59:17.381 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T04:59:17.381 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T04:59:17.381 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T04:59:17.381 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T04:59:17.381 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T04:59:17.381 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T04:59:17.381 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T04:59:17.381 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T04:59:17.381 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T04:59:17.381 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T04:59:17.381 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T04:59:17.381 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T04:59:17.382 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T04:59:17.382 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T04:59:17.382 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T04:59:17.382 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T04:59:17.382 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T04:59:17.382 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T04:59:17.382 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T04:59:17.382 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T04:59:17.382 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T04:59:17.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T04:59:17.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T04:59:17.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T04:59:17.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T04:59:17.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T04:59:17.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T04:59:17.385 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T04:59:17.386 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T04:59:17.386 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T04:59:17.386 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T04:59:17.386 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T04:59:17.386 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T04:59:17.386 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T04:59:17.386 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T04:59:17.386 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T04:59:17.386 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T04:59:17.386 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T04:59:17.386 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T04:59:17.386 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T04:59:17.387 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T04:59:17.387 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T04:59:17.387 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T04:59:17.387 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T04:59:17.387 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T04:59:17.387 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T04:59:17.387 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T04:59:17.387 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T04:59:17.387 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T04:59:17.387 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T04:59:17.387 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T04:59:17.387 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T04:59:17.387 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T04:59:17.387 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T04:59:17.388 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T04:59:17.388 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T04:59:17.388 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T04:59:17.388 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T04:59:17.388 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T04:59:17.388 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T04:59:17.388 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T04:59:17.388 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T04:59:17.388 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T04:59:17.388 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T04:59:17.388 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T04:59:17.388 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T04:59:17.388 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T04:59:17.388 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T04:59:17.389 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T04:59:17.389 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T04:59:17.389 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T04:59:17.389 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T04:59:17.389 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T04:59:17.389 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T04:59:17.389 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T04:59:17.389 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T04:59:17.389 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T04:59:17.389 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T04:59:17.389 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T04:59:17.389 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T04:59:17.389 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:19.0.0-3568.g165a15e5.el9.x86_64 149/164 2024-05-09T04:59:17.389 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T04:59:17.390 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:19.0.0-3568.g165a15e5.el9.x86_64 151/164 2024-05-09T04:59:17.390 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T04:59:17.390 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T04:59:17.390 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T04:59:17.390 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T04:59:17.390 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T04:59:17.390 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T04:59:17.390 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T04:59:17.390 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T04:59:17.390 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T04:59:17.390 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T04:59:17.390 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T04:59:17.390 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T04:59:20.087 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T04:59:20.087 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:20.087 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-05-09T04:59:20.087 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T04:59:20.087 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T04:59:20.087 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T04:59:20.087 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T04:59:20.087 INFO:teuthology.orchestra.run.smithi114.stdout: librados2-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.087 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.088 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T04:59:20.088 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T04:59:20.088 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-05-09T04:59:20.088 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T04:59:20.088 INFO:teuthology.orchestra.run.smithi114.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T04:59:20.088 INFO:teuthology.orchestra.run.smithi114.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T04:59:20.088 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T04:59:20.088 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.088 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.088 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.088 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.088 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:20.089 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.089 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.089 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.089 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:20.089 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:20.089 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:20.089 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:20.089 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:20.089 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.089 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.089 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:20.089 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.090 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.090 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.090 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:20.090 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:20.090 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-mirror-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.090 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-shell-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.090 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-top-2:19.0.0-3568.g165a15e5.el9.noarch 2024-05-09T04:59:20.090 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T04:59:20.090 INFO:teuthology.orchestra.run.smithi114.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T04:59:20.090 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T04:59:20.090 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T04:59:20.091 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T04:59:20.091 INFO:teuthology.orchestra.run.smithi114.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T04:59:20.091 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T04:59:20.091 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T04:59:20.091 INFO:teuthology.orchestra.run.smithi114.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T04:59:20.091 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T04:59:20.091 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T04:59:20.091 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.091 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.091 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.091 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T04:59:20.092 INFO:teuthology.orchestra.run.smithi114.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T04:59:20.092 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T04:59:20.092 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T04:59:20.092 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T04:59:20.092 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.092 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.092 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T04:59:20.092 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.092 INFO:teuthology.orchestra.run.smithi114.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T04:59:20.092 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T04:59:20.092 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T04:59:20.092 INFO:teuthology.orchestra.run.smithi114.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T04:59:20.093 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T04:59:20.093 INFO:teuthology.orchestra.run.smithi114.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T04:59:20.093 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T04:59:20.093 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T04:59:20.093 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T04:59:20.093 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T04:59:20.093 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T04:59:20.093 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T04:59:20.093 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T04:59:20.093 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T04:59:20.093 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T04:59:20.093 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T04:59:20.093 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T04:59:20.094 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T04:59:20.094 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T04:59:20.094 INFO:teuthology.orchestra.run.smithi114.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T04:59:20.094 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T04:59:20.094 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T04:59:20.094 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T04:59:20.094 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.094 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.094 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.095 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T04:59:20.095 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T04:59:20.095 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T04:59:20.095 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T04:59:20.095 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T04:59:20.095 INFO:teuthology.orchestra.run.smithi114.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T04:59:20.095 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T04:59:20.095 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T04:59:20.095 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T04:59:20.095 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T04:59:20.096 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T04:59:20.096 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T04:59:20.096 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T04:59:20.096 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T04:59:20.096 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T04:59:20.096 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T04:59:20.096 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T04:59:20.096 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T04:59:20.096 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T04:59:20.096 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T04:59:20.096 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T04:59:20.097 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T04:59:20.097 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T04:59:20.097 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T04:59:20.097 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T04:59:20.097 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T04:59:20.097 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T04:59:20.097 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T04:59:20.097 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T04:59:20.097 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T04:59:20.097 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T04:59:20.097 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T04:59:20.097 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T04:59:20.098 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T04:59:20.098 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T04:59:20.098 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T04:59:20.098 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T04:59:20.098 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T04:59:20.098 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T04:59:20.098 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T04:59:20.098 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.098 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.098 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T04:59:20.098 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T04:59:20.099 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T04:59:20.099 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.099 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T04:59:20.099 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T04:59:20.099 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T04:59:20.099 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T04:59:20.099 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T04:59:20.099 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T04:59:20.099 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T04:59:20.099 INFO:teuthology.orchestra.run.smithi114.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T04:59:20.099 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T04:59:20.099 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T04:59:20.099 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T04:59:20.100 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T04:59:20.100 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T04:59:20.100 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T04:59:20.100 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T04:59:20.100 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.100 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.100 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd-2:19.0.0-3568.g165a15e5.el9.x86_64 2024-05-09T04:59:20.100 INFO:teuthology.orchestra.run.smithi114.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T04:59:20.100 INFO:teuthology.orchestra.run.smithi114.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T04:59:20.100 INFO:teuthology.orchestra.run.smithi114.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T04:59:20.100 INFO:teuthology.orchestra.run.smithi114.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T04:59:20.100 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T04:59:20.101 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T04:59:20.101 INFO:teuthology.orchestra.run.smithi114.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T04:59:20.101 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:20.101 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-05-09T04:59:20.530 DEBUG:teuthology.parallel:result is None 2024-05-09T04:59:20.531 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=165a15e51d08073b5317c443415b0d8a7141b1ab 2024-05-09T04:59:20.672 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T04:59:20.700 INFO:teuthology.orchestra.run.smithi060.stdout:19.0.0-3568.g165a15e5.el9 2024-05-09T04:59:20.701 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3568.g165a15e5.el9 2024-05-09T04:59:20.701 INFO:teuthology.task.install:The correct ceph version 19.0.0-3568.g165a15e5 is installed. 2024-05-09T04:59:20.702 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=165a15e51d08073b5317c443415b0d8a7141b1ab 2024-05-09T04:59:20.918 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T04:59:20.941 INFO:teuthology.orchestra.run.smithi114.stdout:19.0.0-3568.g165a15e5.el9 2024-05-09T04:59:20.941 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3568.g165a15e5.el9 2024-05-09T04:59:20.942 INFO:teuthology.task.install:The correct ceph version 19.0.0-3568.g165a15e5 is installed. 2024-05-09T04:59:20.943 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-09T04:59:20.943 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:20.943 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T04:59:20.977 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:20.977 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T04:59:21.010 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-09T04:59:21.010 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:21.010 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T04:59:21.047 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T04:59:21.116 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:21.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T04:59:21.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T04:59:21.208 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-09T04:59:21.209 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:21.209 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T04:59:21.238 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T04:59:21.305 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:21.305 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T04:59:21.330 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T04:59:21.395 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-09T04:59:21.396 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:21.396 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T04:59:21.421 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T04:59:21.487 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:21.487 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T04:59:21.512 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T04:59:21.576 INFO:teuthology.run_tasks:Running task ceph... 2024-05-09T04:59:21.654 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-09T04:59:21.654 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 777 /var/log/ceph 2024-05-09T04:59:21.656 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /var/log/ceph 2024-05-09T04:59:21.692 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-09T04:59:21.692 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-09T04:59:21.735 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-09T04:59:21.759 INFO:tasks.ceph:Creating extra log directories... 2024-05-09T04:59:21.759 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-09T04:59:21.803 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-09T04:59:21.885 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-05-09T04:59:21.886 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-05-09T04:59:21.886 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-05-09T04:59:21.886 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:21.886 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-05-09T04:59:21.907 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-05-09T04:59:22.023 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-05-09T04:59:22.050 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:22.050 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-05-09T04:59:22.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-05-09T04:59:22.177 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-05-09T04:59:22.206 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-05-09T04:59:22.206 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-09T04:59:22.207 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi/7699557', 'branch': 'wip-63259-3', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/backtrace}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7699557', 'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-63259-3'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'sha1': '165a15e51d08073b5317c443415b0d8a7141b1ab'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '165a15e51d08073b5317c443415b0d8a7141b1ab'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-63259-qa-fix-1', 'sha1': 'a2e1088fa282408acd6a8a29f9417b763c0a8663'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 9329, 'sha1': '165a15e51d08073b5317c443415b0d8a7141b1ab', 'sleep_before_teardown': 0, 'subset': '712/1024', 'suite': 'fs', 'suite_branch': 'wip-63259-qa-fix-1', 'suite_path': '/home/teuthworker/src/github.com_vshankar_ceph_a2e1088fa282408acd6a8a29f9417b763c0a8663/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/vshankar/ceph.git', 'suite_sha1': 'a2e1088fa282408acd6a8a29f9417b763c0a8663', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi060.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDXKlNmw7xjovvOPA5uoX35oogqy+fhjoG7qlpocOBTDo0OuCLVF4iqXfTNQm2BHg0xoBUU7UPRyEg01wz44wcU=', 'smithi114.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIuN6mPI387qZJmyDmYmuGL/2LfVuKpBV7FADGpOtM1VFIMlHQuH6yKHxSB1OYXGM+A1VzzuYysAGwLIo0zDxDA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '165a15e51d08073b5317c443415b0d8a7141b1ab'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'sha1': '165a15e51d08073b5317c443415b0d8a7141b1ab', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_backtrace']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '1ae7ad82388e92a475afff437d49054826c019a1', 'timestamp': '2024-05-09_04:42:47', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-05-09T04:59:22.207 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-09T04:59:22.210 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-09T04:59:22.257 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/run/ceph 2024-05-09T04:59:22.265 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m0777 -- /var/run/ceph 2024-05-09T04:59:22.322 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:22.322 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T04:59:22.344 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T04:59:22.344 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_1 2024-05-09T04:59:22.401 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T04:59:22.401 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T04:59:22.401 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-05-09T04:59:22.401 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T04:59:22.401 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T04:59:22.401 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-05-09 04:59:02.273012654 +0000 2024-05-09T04:59:22.401 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-05-09 04:56:59.710207655 +0000 2024-05-09T04:59:22.401 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-05-09 04:56:59.710207655 +0000 2024-05-09T04:59:22.401 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-05-09 04:56:59.710207655 +0000 2024-05-09T04:59:22.401 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T04:59:22.466 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-05-09T04:59:22.466 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-05-09T04:59:22.466 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000100619 s, 5.1 MB/s 2024-05-09T04:59:22.467 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T04:59:22.524 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_2 2024-05-09T04:59:22.580 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T04:59:22.580 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T04:59:22.580 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-05-09T04:59:22.580 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T04:59:22.580 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T04:59:22.580 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-05-09 04:59:02.274012644 +0000 2024-05-09T04:59:22.580 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-05-09 04:57:00.051204448 +0000 2024-05-09T04:59:22.580 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-05-09 04:57:00.051204448 +0000 2024-05-09T04:59:22.580 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-05-09 04:57:00.051204448 +0000 2024-05-09T04:59:22.581 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T04:59:22.643 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-05-09T04:59:22.644 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-05-09T04:59:22.644 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000101979 s, 5.0 MB/s 2024-05-09T04:59:22.645 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T04:59:22.701 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_3 2024-05-09T04:59:22.758 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T04:59:22.758 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T04:59:22.758 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-05-09T04:59:22.758 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T04:59:22.758 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T04:59:22.758 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-05-09 04:59:02.274012644 +0000 2024-05-09T04:59:22.758 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-05-09 04:57:00.373201421 +0000 2024-05-09T04:59:22.758 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-05-09 04:57:00.373201421 +0000 2024-05-09T04:59:22.758 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-05-09 04:57:00.373201421 +0000 2024-05-09T04:59:22.759 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T04:59:22.821 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-05-09T04:59:22.822 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-05-09T04:59:22.822 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 9.6794e-05 s, 5.3 MB/s 2024-05-09T04:59:22.823 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T04:59:22.880 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_4 2024-05-09T04:59:22.936 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T04:59:22.936 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T04:59:22.936 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-09T04:59:22.936 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T04:59:22.936 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T04:59:22.936 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-05-09 04:59:02.274012644 +0000 2024-05-09T04:59:22.936 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-05-09 04:57:00.717198186 +0000 2024-05-09T04:59:22.936 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-05-09 04:57:00.717198186 +0000 2024-05-09T04:59:22.936 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-05-09 04:57:00.717198186 +0000 2024-05-09T04:59:22.937 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T04:59:23.000 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-05-09T04:59:23.000 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-05-09T04:59:23.000 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 8.4182e-05 s, 6.1 MB/s 2024-05-09T04:59:23.001 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T04:59:23.058 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T04:59:23.058 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:23.058 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T04:59:23.077 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T04:59:23.077 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-05-09T04:59:23.135 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T04:59:23.136 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T04:59:23.136 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-05-09T04:59:23.136 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T04:59:23.136 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T04:59:23.136 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-05-09 04:59:12.195046005 +0000 2024-05-09T04:59:23.136 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-05-09 04:56:54.621483967 +0000 2024-05-09T04:59:23.136 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-05-09 04:56:54.621483967 +0000 2024-05-09T04:59:23.136 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-05-09 04:56:54.621483967 +0000 2024-05-09T04:59:23.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T04:59:23.203 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-05-09T04:59:23.203 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-05-09T04:59:23.203 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000210404 s, 2.4 MB/s 2024-05-09T04:59:23.204 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T04:59:23.262 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-05-09T04:59:23.322 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T04:59:23.322 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T04:59:23.322 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-09T04:59:23.322 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T04:59:23.322 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T04:59:23.322 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-05-09 04:59:12.196045983 +0000 2024-05-09T04:59:23.322 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-05-09 04:56:54.976476316 +0000 2024-05-09T04:59:23.322 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-05-09 04:56:54.976476316 +0000 2024-05-09T04:59:23.322 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-05-09 04:56:54.976476316 +0000 2024-05-09T04:59:23.323 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T04:59:23.387 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-05-09T04:59:23.387 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-05-09T04:59:23.387 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000228734 s, 2.2 MB/s 2024-05-09T04:59:23.389 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T04:59:23.446 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-05-09T04:59:23.502 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T04:59:23.502 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T04:59:23.503 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-05-09T04:59:23.503 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T04:59:23.503 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T04:59:23.503 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-05-09 04:59:12.196045983 +0000 2024-05-09T04:59:23.503 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-05-09 04:56:55.324468816 +0000 2024-05-09T04:59:23.503 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-05-09 04:56:55.324468816 +0000 2024-05-09T04:59:23.503 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-05-09 04:56:55.324468816 +0000 2024-05-09T04:59:23.503 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T04:59:23.567 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-05-09T04:59:23.567 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-05-09T04:59:23.567 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000215536 s, 2.4 MB/s 2024-05-09T04:59:23.569 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T04:59:23.626 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-05-09T04:59:23.682 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T04:59:23.682 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T04:59:23.682 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-09T04:59:23.682 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T04:59:23.682 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T04:59:23.682 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-05-09 04:59:12.196045983 +0000 2024-05-09T04:59:23.682 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-05-09 04:56:55.655461682 +0000 2024-05-09T04:59:23.682 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-05-09 04:56:55.655461682 +0000 2024-05-09T04:59:23.682 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-05-09 04:56:55.655461682 +0000 2024-05-09T04:59:23.683 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T04:59:23.745 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-05-09T04:59:23.745 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-05-09T04:59:23.745 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000207285 s, 2.5 MB/s 2024-05-09T04:59:23.747 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T04:59:23.803 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T04:59:23.803 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi114.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-09T04:59:23.803 INFO:tasks.ceph:Generating config... 2024-05-09T04:59:23.804 INFO:tasks.ceph:[client] client mount timeout = 600 2024-05-09T04:59:23.804 INFO:tasks.ceph:[client] debug client = 20 2024-05-09T04:59:23.804 INFO:tasks.ceph:[client] debug ms = 1 2024-05-09T04:59:23.804 INFO:tasks.ceph:[client] pid file = 2024-05-09T04:59:23.804 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-05-09T04:59:23.805 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-05-09T04:59:23.805 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-05-09T04:59:23.805 INFO:tasks.ceph:[mds] debug mds = 20 2024-05-09T04:59:23.805 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-05-09T04:59:23.805 INFO:tasks.ceph:[mds] debug ms = 1 2024-05-09T04:59:23.805 INFO:tasks.ceph:[mds] mds debug frag = True 2024-05-09T04:59:23.805 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-05-09T04:59:23.805 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-05-09T04:59:23.805 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-05-09T04:59:23.805 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-05-09T04:59:23.805 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-05-09T04:59:23.805 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-05-09T04:59:23.805 INFO:tasks.ceph:[mgr] debug client = 20 2024-05-09T04:59:23.805 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-09T04:59:23.805 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-09T04:59:23.805 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-09T04:59:23.805 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-09T04:59:23.806 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-09T04:59:23.806 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-05-09T04:59:23.806 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-05-09T04:59:23.806 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-09T04:59:23.806 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-09T04:59:23.806 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-05-09T04:59:23.806 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-05-09T04:59:23.806 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-09T04:59:23.806 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-09T04:59:23.806 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-05-09T04:59:23.806 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-09T04:59:23.806 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-09T04:59:23.806 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-09T04:59:23.806 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-09T04:59:23.807 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-09T04:59:23.807 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-05-09T04:59:23.807 INFO:tasks.ceph:Setting up mon.a... 2024-05-09T04:59:23.807 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-05-09T04:59:23.864 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/ceph.keyring 2024-05-09T04:59:23.867 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-05-09T04:59:23.912 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-09T04:59:23.937 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.60'), ('mon.b', '172.21.15.114'), ('mon.c', '[v2:172.21.15.114:3301,v1:172.21.15.114:6790]')] 2024-05-09T04:59:23.938 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.60,172.21.15.114,[v2:172.21.15.114:3301,v1:172.21.15.114:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-05-09T04:59:23.938 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:23.939 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-09T04:59:23.993 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.60 --add b 172.21.15.114 --addv c '[v2:172.21.15.114:3301,v1:172.21.15.114:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-05-09T04:59:24.081 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-05-09T04:59:24.081 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: generated fsid 0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2 2024-05-09T04:59:24.082 INFO:teuthology.orchestra.run.smithi060.stdout:setting min_mon_release = quincy 2024-05-09T04:59:24.082 INFO:teuthology.orchestra.run.smithi060.stdout:epoch 0 2024-05-09T04:59:24.082 INFO:teuthology.orchestra.run.smithi060.stdout:fsid 0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2 2024-05-09T04:59:24.082 INFO:teuthology.orchestra.run.smithi060.stdout:last_changed 2024-05-09T04:59:24.082324+0000 2024-05-09T04:59:24.082 INFO:teuthology.orchestra.run.smithi060.stdout:created 2024-05-09T04:59:24.082324+0000 2024-05-09T04:59:24.082 INFO:teuthology.orchestra.run.smithi060.stdout:min_mon_release 17 (quincy) 2024-05-09T04:59:24.082 INFO:teuthology.orchestra.run.smithi060.stdout:election_strategy: 1 2024-05-09T04:59:24.082 INFO:teuthology.orchestra.run.smithi060.stdout:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-05-09T04:59:24.082 INFO:teuthology.orchestra.run.smithi060.stdout:1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-05-09T04:59:24.082 INFO:teuthology.orchestra.run.smithi060.stdout:2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-05-09T04:59:24.082 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-05-09T04:59:24.083 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-09T04:59:24.099 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2... 2024-05-09T04:59:24.100 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-09T04:59:24.141 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-09T04:59:24.183 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2024-05-09T04:59:24.183 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2024-05-09T04:59:24.184 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-09T04:59:24.184 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = cephx 2024-05-09T04:59:24.184 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.184 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2024-05-09T04:59:24.184 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.184 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2024-05-09T04:59:24.184 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.184 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2024-05-09T04:59:24.184 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2024-05-09T04:59:24.184 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.184 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2024-05-09T04:59:24.184 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2024-05-09T04:59:24.184 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.185 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-09T04:59:24.185 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2024-05-09T04:59:24.185 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.185 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2024-05-09T04:59:24.185 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-09T04:59:24.185 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.185 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2024-05-09T04:59:24.185 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.185 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2024-05-09T04:59:24.185 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2024-05-09T04:59:24.185 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2024-05-09T04:59:24.185 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2024-05-09T04:59:24.185 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2024-05-09T04:59:24.185 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2024-05-09T04:59:24.186 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2024-05-09T04:59:24.186 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-09T04:59:24.186 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-09T04:59:24.186 INFO:teuthology.orchestra.run.smithi060.stdout: mon_allow_pool_size_one = true 2024-05-09T04:59:24.186 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.186 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-09T04:59:24.186 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.186 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2024-05-09T04:59:24.186 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.186 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2024-05-09T04:59:24.186 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.186 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2024-05-09T04:59:24.186 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2024-05-09T04:59:24.186 INFO:teuthology.orchestra.run.smithi060.stdout: mon health detail to clog = false 2024-05-09T04:59:24.187 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = "172.21.15.60,172.21.15.114,[v2:172.21.15.114:3301,v1:172.21.15.114:6790]" 2024-05-09T04:59:24.187 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on skipped message = False 2024-05-09T04:59:24.187 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = 0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2 2024-05-09T04:59:24.187 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.187 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2024-05-09T04:59:24.187 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2024-05-09T04:59:24.187 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.187 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2024-05-09T04:59:24.188 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 600 2024-05-09T04:59:24.188 INFO:teuthology.orchestra.run.smithi060.stdout: osd mclock profile = high_recovery_ops 2024-05-09T04:59:24.188 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.188 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2024-05-09T04:59:24.188 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2024-05-09T04:59:24.188 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.188 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2024-05-09T04:59:24.188 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2024-05-09T04:59:24.188 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2024-05-09T04:59:24.188 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug trim objects = true 2024-05-09T04:59:24.188 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.188 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2024-05-09T04:59:24.189 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2024-05-09T04:59:24.189 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.189 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2024-05-09T04:59:24.189 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.189 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2024-05-09T04:59:24.189 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.189 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2024-05-09T04:59:24.189 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.189 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2024-05-09T04:59:24.189 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2024-05-09T04:59:24.189 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.189 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2024-05-09T04:59:24.189 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2024-05-09T04:59:24.190 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore block size = 96636764160 2024-05-09T04:59:24.190 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore fsck on mount = True 2024-05-09T04:59:24.190 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluefs = 20 2024-05-09T04:59:24.190 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluestore = 20 2024-05-09T04:59:24.190 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-05-09T04:59:24.190 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 20 2024-05-09T04:59:24.190 INFO:teuthology.orchestra.run.smithi060.stdout: debug rocksdb = 10 2024-05-09T04:59:24.190 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd backfillfull_ratio = 0.85 2024-05-09T04:59:24.190 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd full ratio = 0.9 2024-05-09T04:59:24.190 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd nearfull ratio = 0.8 2024-05-09T04:59:24.190 INFO:teuthology.orchestra.run.smithi060.stdout: osd failsafe full ratio = 0.95 2024-05-09T04:59:24.190 INFO:teuthology.orchestra.run.smithi060.stdout: osd objectstore = bluestore 2024-05-09T04:59:24.190 INFO:teuthology.orchestra.run.smithi060.stdout: osd op complaint time = 180 2024-05-09T04:59:24.190 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.191 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2024-05-09T04:59:24.191 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-05-09T04:59:24.191 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2024-05-09T04:59:24.191 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-05-09T04:59:24.191 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-05-09T04:59:24.191 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-05-09T04:59:24.191 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-05-09T04:59:24.191 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2024-05-09T04:59:24.191 INFO:teuthology.orchestra.run.smithi060.stdout: debug client = 20 2024-05-09T04:59:24.191 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.191 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2024-05-09T04:59:24.191 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-05-09T04:59:24.191 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-05-09T04:59:24.192 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 20 2024-05-09T04:59:24.192 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-05-09T04:59:24.192 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2024-05-09T04:59:24.192 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2024-05-09T04:59:24.192 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-05-09T04:59:24.192 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2024-05-09T04:59:24.192 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2024-05-09T04:59:24.192 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-05-09T04:59:24.192 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.192 INFO:teuthology.orchestra.run.smithi060.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-09T04:59:24.192 INFO:teuthology.orchestra.run.smithi060.stdout: auth mon ticket ttl = 660 # 11m 2024-05-09T04:59:24.192 INFO:teuthology.orchestra.run.smithi060.stdout: auth service ticket ttl = 240 # 4m 2024-05-09T04:59:24.192 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.192 INFO:teuthology.orchestra.run.smithi060.stdout: # don't complain about insecure global_id in the test suite 2024-05-09T04:59:24.193 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-09T04:59:24.193 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-09T04:59:24.193 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.193 INFO:teuthology.orchestra.run.smithi060.stdout: # 1m isn't quite enough 2024-05-09T04:59:24.193 INFO:teuthology.orchestra.run.smithi060.stdout: mon_down_mkfs_grace = 2m 2024-05-09T04:59:24.193 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.193 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_filestore_osds = false 2024-05-09T04:59:24.193 INFO:teuthology.orchestra.run.smithi060.stdout: mon down mkfs grace = 300 2024-05-09T04:59:24.193 INFO:teuthology.orchestra.run.smithi060.stdout: mon op complaint time = 120 2024-05-09T04:59:24.193 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:24.193 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2024-05-09T04:59:24.193 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2024-05-09T04:59:24.193 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2024-05-09T04:59:24.193 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2024-05-09T04:59:24.193 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-09T04:59:24.193 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-09T04:59:24.194 INFO:teuthology.orchestra.run.smithi060.stdout: client mount timeout = 600 2024-05-09T04:59:24.194 INFO:teuthology.orchestra.run.smithi060.stdout: debug client = 20 2024-05-09T04:59:24.194 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-05-09T04:59:24.194 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = "" 2024-05-09T04:59:24.194 INFO:teuthology.orchestra.run.smithi060.stdout: rados mon op timeout = 900 2024-05-09T04:59:24.194 INFO:teuthology.orchestra.run.smithi060.stdout: rados osd op timeout = 900 2024-05-09T04:59:24.194 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2024-05-09T04:59:24.194 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.b] 2024-05-09T04:59:24.194 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.c] 2024-05-09T04:59:24.194 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.a] 2024-05-09T04:59:24.194 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.c] 2024-05-09T04:59:24.194 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.b] 2024-05-09T04:59:24.194 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.d] 2024-05-09T04:59:24.194 INFO:teuthology.orchestra.run.smithi060.stdout:[mds] 2024-05-09T04:59:24.194 INFO:teuthology.orchestra.run.smithi060.stdout: debug mds = 20 2024-05-09T04:59:24.195 INFO:teuthology.orchestra.run.smithi060.stdout: debug mds balancer = 20 2024-05-09T04:59:24.195 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-05-09T04:59:24.195 INFO:teuthology.orchestra.run.smithi060.stdout: mds debug frag = True 2024-05-09T04:59:24.195 INFO:teuthology.orchestra.run.smithi060.stdout: mds debug scatterstat = True 2024-05-09T04:59:24.195 INFO:teuthology.orchestra.run.smithi060.stdout: mds op complaint time = 180 2024-05-09T04:59:24.195 INFO:teuthology.orchestra.run.smithi060.stdout: mds verify scatter = True 2024-05-09T04:59:24.195 INFO:teuthology.orchestra.run.smithi060.stdout: osd op complaint time = 180 2024-05-09T04:59:24.195 INFO:teuthology.orchestra.run.smithi060.stdout: rados mon op timeout = 900 2024-05-09T04:59:24.195 INFO:teuthology.orchestra.run.smithi060.stdout: rados osd op timeout = 900 2024-05-09T04:59:24.196 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-05-09T04:59:24.197 INFO:teuthology.orchestra.run.smithi114.stdout: chdir = "" 2024-05-09T04:59:24.197 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-09T04:59:24.197 INFO:teuthology.orchestra.run.smithi114.stdout: auth supported = cephx 2024-05-09T04:59:24.197 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.197 INFO:teuthology.orchestra.run.smithi114.stdout: filestore xattr use omap = true 2024-05-09T04:59:24.197 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.197 INFO:teuthology.orchestra.run.smithi114.stdout: mon clock drift allowed = 1.000 2024-05-09T04:59:24.197 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.197 INFO:teuthology.orchestra.run.smithi114.stdout: osd crush chooseleaf type = 0 2024-05-09T04:59:24.197 INFO:teuthology.orchestra.run.smithi114.stdout: auth debug = true 2024-05-09T04:59:24.197 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.197 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on old message = true 2024-05-09T04:59:24.197 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on bug = true 2024-05-09T04:59:24.197 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.198 INFO:teuthology.orchestra.run.smithi114.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-09T04:59:24.198 INFO:teuthology.orchestra.run.smithi114.stdout: mon pg warn max object skew = 0 2024-05-09T04:59:24.198 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.198 INFO:teuthology.orchestra.run.smithi114.stdout: # disable pg_autoscaler by default for new pools 2024-05-09T04:59:24.198 INFO:teuthology.orchestra.run.smithi114.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-09T04:59:24.198 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.198 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default size = 2 2024-05-09T04:59:24.198 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.198 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow primary affinity = true 2024-05-09T04:59:24.198 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow pg remap = true 2024-05-09T04:59:24.198 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on legacy crush tunables = false 2024-05-09T04:59:24.198 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on crush straw calc version zero = false 2024-05-09T04:59:24.198 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on no sortbitwise = false 2024-05-09T04:59:24.198 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on osd down out interval zero = false 2024-05-09T04:59:24.199 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on too few osds = false 2024-05-09T04:59:24.199 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-09T04:59:24.199 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-09T04:59:24.199 INFO:teuthology.orchestra.run.smithi114.stdout: mon_allow_pool_size_one = true 2024-05-09T04:59:24.199 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.199 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-09T04:59:24.199 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.199 INFO:teuthology.orchestra.run.smithi114.stdout: osd default data pool replay window = 5 2024-05-09T04:59:24.199 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.199 INFO:teuthology.orchestra.run.smithi114.stdout: mon allow pool delete = true 2024-05-09T04:59:24.199 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.199 INFO:teuthology.orchestra.run.smithi114.stdout: mon cluster log file level = debug 2024-05-09T04:59:24.199 INFO:teuthology.orchestra.run.smithi114.stdout: debug asserts on shutdown = true 2024-05-09T04:59:24.199 INFO:teuthology.orchestra.run.smithi114.stdout: mon health detail to clog = false 2024-05-09T04:59:24.199 INFO:teuthology.orchestra.run.smithi114.stdout: mon host = "172.21.15.60,172.21.15.114,[v2:172.21.15.114:3301,v1:172.21.15.114:6790]" 2024-05-09T04:59:24.199 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on skipped message = False 2024-05-09T04:59:24.200 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = 0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2 2024-05-09T04:59:24.200 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.200 INFO:teuthology.orchestra.run.smithi114.stdout:[osd] 2024-05-09T04:59:24.200 INFO:teuthology.orchestra.run.smithi114.stdout: osd journal size = 100 2024-05-09T04:59:24.200 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.200 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub load threshold = 5.0 2024-05-09T04:59:24.200 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub max interval = 600 2024-05-09T04:59:24.200 INFO:teuthology.orchestra.run.smithi114.stdout: osd mclock profile = high_recovery_ops 2024-05-09T04:59:24.200 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.200 INFO:teuthology.orchestra.run.smithi114.stdout: osd recover clone overlap = true 2024-05-09T04:59:24.200 INFO:teuthology.orchestra.run.smithi114.stdout: osd recovery max chunk = 1048576 2024-05-09T04:59:24.200 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.200 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug shutdown = true 2024-05-09T04:59:24.200 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug op order = true 2024-05-09T04:59:24.200 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify stray on activate = true 2024-05-09T04:59:24.200 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug trim objects = true 2024-05-09T04:59:24.201 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.201 INFO:teuthology.orchestra.run.smithi114.stdout: osd open classes on start = true 2024-05-09T04:59:24.201 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug pg log writeout = true 2024-05-09T04:59:24.201 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.201 INFO:teuthology.orchestra.run.smithi114.stdout: osd deep scrub update digest min age = 30 2024-05-09T04:59:24.201 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.201 INFO:teuthology.orchestra.run.smithi114.stdout: osd map max advance = 10 2024-05-09T04:59:24.201 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.201 INFO:teuthology.orchestra.run.smithi114.stdout: journal zero on create = true 2024-05-09T04:59:24.201 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.201 INFO:teuthology.orchestra.run.smithi114.stdout: filestore ondisk finisher threads = 3 2024-05-09T04:59:24.201 INFO:teuthology.orchestra.run.smithi114.stdout: filestore apply finisher threads = 3 2024-05-09T04:59:24.201 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.201 INFO:teuthology.orchestra.run.smithi114.stdout: bdev debug aio = true 2024-05-09T04:59:24.201 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug misdirected ops = true 2024-05-09T04:59:24.201 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore block size = 96636764160 2024-05-09T04:59:24.202 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore fsck on mount = True 2024-05-09T04:59:24.202 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluefs = 20 2024-05-09T04:59:24.202 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluestore = 20 2024-05-09T04:59:24.202 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-05-09T04:59:24.202 INFO:teuthology.orchestra.run.smithi114.stdout: debug osd = 20 2024-05-09T04:59:24.202 INFO:teuthology.orchestra.run.smithi114.stdout: debug rocksdb = 10 2024-05-09T04:59:24.202 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd backfillfull_ratio = 0.85 2024-05-09T04:59:24.202 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd full ratio = 0.9 2024-05-09T04:59:24.202 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd nearfull ratio = 0.8 2024-05-09T04:59:24.202 INFO:teuthology.orchestra.run.smithi114.stdout: osd failsafe full ratio = 0.95 2024-05-09T04:59:24.202 INFO:teuthology.orchestra.run.smithi114.stdout: osd objectstore = bluestore 2024-05-09T04:59:24.202 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2024-05-09T04:59:24.202 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.202 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr] 2024-05-09T04:59:24.202 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-05-09T04:59:24.202 INFO:teuthology.orchestra.run.smithi114.stdout: debug mgr = 20 2024-05-09T04:59:24.203 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-05-09T04:59:24.203 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-05-09T04:59:24.203 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-05-09T04:59:24.203 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-05-09T04:59:24.203 INFO:teuthology.orchestra.run.smithi114.stdout: mgr/telemetry/nag = false 2024-05-09T04:59:24.203 INFO:teuthology.orchestra.run.smithi114.stdout: debug client = 20 2024-05-09T04:59:24.203 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.203 INFO:teuthology.orchestra.run.smithi114.stdout:[mon] 2024-05-09T04:59:24.203 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-05-09T04:59:24.203 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-05-09T04:59:24.203 INFO:teuthology.orchestra.run.smithi114.stdout: debug paxos = 20 2024-05-09T04:59:24.203 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-05-09T04:59:24.203 INFO:teuthology.orchestra.run.smithi114.stdout: mon data avail warn = 5 2024-05-09T04:59:24.203 INFO:teuthology.orchestra.run.smithi114.stdout: mon mgr mkfs grace = 240 2024-05-09T04:59:24.203 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-05-09T04:59:24.203 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd reporter subtree level = osd 2024-05-09T04:59:24.204 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd prime pg temp = true 2024-05-09T04:59:24.204 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-05-09T04:59:24.204 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.204 INFO:teuthology.orchestra.run.smithi114.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-09T04:59:24.204 INFO:teuthology.orchestra.run.smithi114.stdout: auth mon ticket ttl = 660 # 11m 2024-05-09T04:59:24.204 INFO:teuthology.orchestra.run.smithi114.stdout: auth service ticket ttl = 240 # 4m 2024-05-09T04:59:24.204 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.204 INFO:teuthology.orchestra.run.smithi114.stdout: # don't complain about insecure global_id in the test suite 2024-05-09T04:59:24.204 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-09T04:59:24.204 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-09T04:59:24.204 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.204 INFO:teuthology.orchestra.run.smithi114.stdout: # 1m isn't quite enough 2024-05-09T04:59:24.204 INFO:teuthology.orchestra.run.smithi114.stdout: mon_down_mkfs_grace = 2m 2024-05-09T04:59:24.205 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.205 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_filestore_osds = false 2024-05-09T04:59:24.205 INFO:teuthology.orchestra.run.smithi114.stdout: mon down mkfs grace = 300 2024-05-09T04:59:24.205 INFO:teuthology.orchestra.run.smithi114.stdout: mon op complaint time = 120 2024-05-09T04:59:24.205 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-05-09T04:59:24.205 INFO:teuthology.orchestra.run.smithi114.stdout:[client] 2024-05-09T04:59:24.205 INFO:teuthology.orchestra.run.smithi114.stdout: rgw cache enabled = true 2024-05-09T04:59:24.205 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable ops log = true 2024-05-09T04:59:24.205 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable usage log = true 2024-05-09T04:59:24.205 INFO:teuthology.orchestra.run.smithi114.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-09T04:59:24.205 INFO:teuthology.orchestra.run.smithi114.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-09T04:59:24.205 INFO:teuthology.orchestra.run.smithi114.stdout: client mount timeout = 600 2024-05-09T04:59:24.206 INFO:teuthology.orchestra.run.smithi114.stdout: debug client = 20 2024-05-09T04:59:24.206 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-05-09T04:59:24.206 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = "" 2024-05-09T04:59:24.206 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2024-05-09T04:59:24.206 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2024-05-09T04:59:24.206 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.a] 2024-05-09T04:59:24.206 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.b] 2024-05-09T04:59:24.206 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.c] 2024-05-09T04:59:24.206 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.a] 2024-05-09T04:59:24.206 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.c] 2024-05-09T04:59:24.206 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.b] 2024-05-09T04:59:24.206 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.d] 2024-05-09T04:59:24.206 INFO:teuthology.orchestra.run.smithi114.stdout:[mds] 2024-05-09T04:59:24.206 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds = 20 2024-05-09T04:59:24.207 INFO:teuthology.orchestra.run.smithi114.stdout: debug mds balancer = 20 2024-05-09T04:59:24.207 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-05-09T04:59:24.207 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug frag = True 2024-05-09T04:59:24.207 INFO:teuthology.orchestra.run.smithi114.stdout: mds debug scatterstat = True 2024-05-09T04:59:24.207 INFO:teuthology.orchestra.run.smithi114.stdout: mds op complaint time = 180 2024-05-09T04:59:24.207 INFO:teuthology.orchestra.run.smithi114.stdout: mds verify scatter = True 2024-05-09T04:59:24.207 INFO:teuthology.orchestra.run.smithi114.stdout: osd op complaint time = 180 2024-05-09T04:59:24.207 INFO:teuthology.orchestra.run.smithi114.stdout: rados mon op timeout = 900 2024-05-09T04:59:24.207 INFO:teuthology.orchestra.run.smithi114.stdout: rados osd op timeout = 900 2024-05-09T04:59:24.207 INFO:tasks.ceph:Creating admin key on mon.a... 2024-05-09T04:59:24.207 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-05-09T04:59:24.294 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-09T04:59:24.294 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:24.294 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-05-09T04:59:24.310 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:24.310 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-05-09T04:59:24.366 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi060.front.sepia.ceph.com 2024-05-09T04:59:24.366 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:24.366 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-09T04:59:24.366 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-09T04:59:24.442 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:24.442 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-09T04:59:24.498 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T04:59:24.498 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:24.498 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-09T04:59:24.498 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-09T04:59:24.533 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:24.533 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-09T04:59:24.589 INFO:tasks.ceph:Setting up mon nodes... 2024-05-09T04:59:24.590 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-09T04:59:24.590 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-05-09T04:59:24.643 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-05-09T04:59:24.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-05-09T04:59:24.702 INFO:teuthology.orchestra.run.smithi114.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-05-09T04:59:24.704 INFO:tasks.ceph:Setting up mds nodes... 2024-05-09T04:59:24.705 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-05-09T04:59:24.757 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-05-09T04:59:24.760 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-05-09T04:59:24.785 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-05-09T04:59:24.878 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-05-09T04:59:24.880 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-05-09T04:59:24.906 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-05-09T04:59:24.957 INFO:teuthology.orchestra.run.smithi114.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-05-09T04:59:24.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-05-09T04:59:24.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-05-09T04:59:25.075 INFO:teuthology.orchestra.run.smithi114.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-05-09T04:59:25.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-05-09T04:59:25.102 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-09T04:59:25.102 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-09T04:59:25.145 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-05-09T04:59:25.158 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-05-09T04:59:25.241 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-05-09T04:59:25.255 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-09T04:59:25.295 INFO:teuthology.orchestra.run.smithi114.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-05-09T04:59:25.308 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-05-09T04:59:25.389 INFO:teuthology.orchestra.run.smithi114.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-05-09T04:59:25.402 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-09T04:59:25.402 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi114.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-09T04:59:25.402 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-05-09T04:59:25.428 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T04:59:25.428 INFO:tasks.ceph:role: osd.0 2024-05-09T04:59:25.428 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com 2024-05-09T04:59:25.428 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-09T04:59:25.494 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T04:59:25.494 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T04:59:25.494 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T04:59:25.494 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T04:59:25.494 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T04:59:25.494 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-05-09T04:59:25.494 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T04:59:25.494 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T04:59:25.494 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T04:59:25.494 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T04:59:25.526 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-05-09T04:59:25.528 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-05-09T04:59:25.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-05-09T04:59:25.599 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-05-09T04:59:25.670 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-05-09T04:59:25.738 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T04:59:25.738 INFO:tasks.ceph:role: osd.1 2024-05-09T04:59:25.738 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com 2024-05-09T04:59:25.738 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-09T04:59:25.804 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T04:59:25.804 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T04:59:25.804 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T04:59:25.804 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T04:59:25.804 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T04:59:25.804 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-05-09T04:59:25.804 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T04:59:25.804 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T04:59:25.804 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T04:59:25.804 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T04:59:25.837 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-05-09T04:59:25.839 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-05-09T04:59:25.839 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-05-09T04:59:25.909 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-05-09T04:59:25.979 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-05-09T04:59:26.045 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T04:59:26.045 INFO:tasks.ceph:role: osd.2 2024-05-09T04:59:26.045 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com 2024-05-09T04:59:26.045 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-09T04:59:26.110 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T04:59:26.110 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T04:59:26.110 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T04:59:26.110 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T04:59:26.110 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T04:59:26.110 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-05-09T04:59:26.110 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T04:59:26.110 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T04:59:26.110 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T04:59:26.110 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T04:59:26.143 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-05-09T04:59:26.144 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-05-09T04:59:26.145 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-05-09T04:59:26.215 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-05-09T04:59:26.285 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-05-09T04:59:26.351 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T04:59:26.351 INFO:tasks.ceph:role: osd.3 2024-05-09T04:59:26.351 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi060.front.sepia.ceph.com 2024-05-09T04:59:26.352 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-09T04:59:26.416 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T04:59:26.416 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T04:59:26.417 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T04:59:26.417 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T04:59:26.417 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T04:59:26.417 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-05-09T04:59:26.417 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T04:59:26.417 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T04:59:26.417 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T04:59:26.417 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T04:59:26.448 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-05-09T04:59:26.450 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-05-09T04:59:26.451 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-05-09T04:59:26.521 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-05-09T04:59:26.591 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T04:59:26.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:26.685+0000 7f816f213640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-05-09T04:59:26.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:26.685+0000 7f816f213640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-05-09T04:59:26.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:26.685+0000 7f816f213640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-09T04:59:26.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:26.685+0000 7f816f213640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-09T04:59:26.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:26.685+0000 7f816f213640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-09T04:59:26.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:26.685+0000 7f816f213640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-05-09T04:59:29.541 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-09T04:59:29.572 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T04:59:29.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:29.663+0000 7fecfe814640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-05-09T04:59:29.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:29.663+0000 7fecfe814640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-05-09T04:59:29.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:29.663+0000 7fecfe814640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-09T04:59:29.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:29.663+0000 7fecfe814640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-09T04:59:29.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:29.664+0000 7fecfe814640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-09T04:59:29.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:29.664+0000 7fecfe814640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-05-09T04:59:32.507 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-09T04:59:32.537 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T04:59:32.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:32.630+0000 7ff7cd54c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-05-09T04:59:32.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:32.630+0000 7ff7cd54c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-05-09T04:59:32.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:32.630+0000 7ff7cd54c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-09T04:59:32.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:32.630+0000 7ff7cd54c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-09T04:59:32.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:32.630+0000 7ff7cd54c640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-09T04:59:32.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:32.630+0000 7ff7cd54c640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-05-09T04:59:35.485 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-09T04:59:35.515 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T04:59:35.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:35.607+0000 7fa793615640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-05-09T04:59:35.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:35.607+0000 7fa793615640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-05-09T04:59:35.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:35.607+0000 7fa793615640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-09T04:59:35.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:35.608+0000 7fa793615640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-09T04:59:35.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:35.608+0000 7fa793615640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-09T04:59:35.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:35.608+0000 7fa793615640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-05-09T04:59:38.457 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-09T04:59:38.487 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-05-09T04:59:38.517 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T04:59:38.517 INFO:tasks.ceph:role: osd.4 2024-05-09T04:59:38.517 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T04:59:38.517 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-09T04:59:38.593 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T04:59:38.593 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T04:59:38.593 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T04:59:38.593 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T04:59:38.593 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T04:59:38.593 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-05-09T04:59:38.593 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T04:59:38.594 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T04:59:38.594 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T04:59:38.594 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T04:59:39.103 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-05-09T04:59:39.105 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-05-09T04:59:39.105 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-05-09T04:59:39.239 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-05-09T04:59:39.272 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-05-09T04:59:39.338 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T04:59:39.339 INFO:tasks.ceph:role: osd.5 2024-05-09T04:59:39.339 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T04:59:39.339 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-09T04:59:39.412 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T04:59:39.412 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T04:59:39.412 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T04:59:39.412 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T04:59:39.412 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T04:59:39.412 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-05-09T04:59:39.413 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T04:59:39.413 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T04:59:39.413 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T04:59:39.413 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T04:59:39.865 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-05-09T04:59:39.867 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-05-09T04:59:39.867 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-05-09T04:59:39.952 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-05-09T04:59:39.980 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-05-09T04:59:40.044 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T04:59:40.044 INFO:tasks.ceph:role: osd.6 2024-05-09T04:59:40.045 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T04:59:40.045 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-09T04:59:40.117 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T04:59:40.118 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T04:59:40.118 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T04:59:40.118 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T04:59:40.118 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T04:59:40.118 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-05-09T04:59:40.118 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T04:59:40.118 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T04:59:40.118 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T04:59:40.118 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T04:59:40.611 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-05-09T04:59:40.613 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-05-09T04:59:40.613 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-05-09T04:59:40.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-05-09T04:59:40.792 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-05-09T04:59:40.857 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T04:59:40.858 INFO:tasks.ceph:role: osd.7 2024-05-09T04:59:40.858 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T04:59:40.858 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-09T04:59:40.931 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T04:59:40.931 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T04:59:40.931 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T04:59:40.931 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T04:59:40.931 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T04:59:40.931 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-05-09T04:59:40.931 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T04:59:40.931 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T04:59:40.931 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T04:59:40.931 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T04:59:41.447 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-05-09T04:59:41.449 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-05-09T04:59:41.449 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-05-09T04:59:41.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-05-09T04:59:41.625 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T04:59:41.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:41.733+0000 7fbed6063640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-05-09T04:59:41.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:41.734+0000 7fbed6063640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-05-09T04:59:41.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:41.734+0000 7fbed6063640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-09T04:59:41.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:41.734+0000 7fbed6063640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-09T04:59:41.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:41.734+0000 7fbed6063640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-09T04:59:41.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:41.734+0000 7fbed6063640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-05-09T04:59:44.582 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-09T04:59:44.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T04:59:44.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:44.699+0000 7fc463ae8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-05-09T04:59:44.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:44.699+0000 7fc463ae8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-05-09T04:59:44.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:44.699+0000 7fc463ae8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-09T04:59:44.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:44.699+0000 7fc463ae8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-09T04:59:44.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:44.699+0000 7fc463ae8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-09T04:59:44.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:44.700+0000 7fc463ae8640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-05-09T04:59:47.557 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-09T04:59:47.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T04:59:47.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:47.677+0000 7f50de41a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-05-09T04:59:47.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:47.677+0000 7f50de41a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-05-09T04:59:47.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:47.677+0000 7f50de41a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-09T04:59:47.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:47.677+0000 7f50de41a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-09T04:59:47.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:47.677+0000 7f50de41a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-09T04:59:47.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:47.678+0000 7f50de41a640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-05-09T04:59:50.533 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-09T04:59:50.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T04:59:50.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:50.644+0000 7fa865210640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-05-09T04:59:50.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:50.645+0000 7fa865210640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-05-09T04:59:50.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:50.645+0000 7fa865210640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-09T04:59:50.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:50.645+0000 7fa865210640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-09T04:59:50.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:50.645+0000 7fa865210640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-09T04:59:50.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:50.645+0000 7fa865210640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-05-09T04:59:52.209 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T04:59:52.211 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T04:59:53.497 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-09T04:59:53.526 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-09T04:59:53.526 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:53.526 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-05-09T04:59:53.557 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:53.557 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-05-09T04:59:53.627 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:53.627 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-05-09T04:59:53.696 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:53.697 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-05-09T04:59:53.766 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:53.766 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-05-09T04:59:53.834 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:53.834 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-05-09T04:59:53.901 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:53.901 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-05-09T04:59:53.969 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:53.969 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-05-09T04:59:53.999 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:53.999 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-05-09T04:59:54.068 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:54.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-05-09T04:59:54.136 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:54.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-05-09T04:59:54.204 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:54.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-05-09T04:59:54.272 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:54.272 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-05-09T04:59:54.337 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:54.338 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-05-09T04:59:54.403 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:54.403 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-05-09T04:59:54.422 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:54.422 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-05-09T04:59:54.481 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:54.481 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-05-09T04:59:54.497 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:54.497 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-05-09T04:59:54.554 INFO:tasks.ceph:Adding keys to all mons... 2024-05-09T04:59:54.554 DEBUG:teuthology.orchestra.run.smithi060:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-09T04:59:54.557 DEBUG:teuthology.orchestra.run.smithi114:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-09T04:59:54.597 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.y] 2024-05-09T04:59:54.597 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAsWDxmdBNeJhAAyU0fz/hMt2qp285unqOtlQ== 2024-05-09T04:59:54.597 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.a] 2024-05-09T04:59:54.597 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAsWDxmJBQ1LRAAJfklX7rdjYuv6WIp+8rgMg== 2024-05-09T04:59:54.598 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.c] 2024-05-09T04:59:54.598 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAsWDxmmr9iNBAAyatU9+hjHFV/AFTKpVxMPQ== 2024-05-09T04:59:54.598 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.0] 2024-05-09T04:59:54.598 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAuWDxmOInrKBAAthThp6oxwvbii2eif7krkg== 2024-05-09T04:59:54.598 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.1] 2024-05-09T04:59:54.598 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAxWDxmbDifJxAAhJ5xGS3YfkNkYuJYF7MiIg== 2024-05-09T04:59:54.598 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.2] 2024-05-09T04:59:54.598 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQA0WDxmrlOkJRAAf6eB9x9UcDCilfbWBIMCGQ== 2024-05-09T04:59:54.599 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.3] 2024-05-09T04:59:54.599 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQA3WDxm2ItKJBAAO4FLP3sH/POba/k7DGFXqA== 2024-05-09T04:59:54.599 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.x] 2024-05-09T04:59:54.599 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAsWDxmuRPmKRAA8uT8sJ8VL1XVxa6c3RTj6g== 2024-05-09T04:59:54.599 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.b] 2024-05-09T04:59:54.599 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAsWDxmUN0fORAAaKZA9GlWlOfJaY/SYAXNlw== 2024-05-09T04:59:54.599 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.d] 2024-05-09T04:59:54.600 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAtWDxms0CIBBAAsHfza+C91qgfGjrczfoGeg== 2024-05-09T04:59:54.600 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.4] 2024-05-09T04:59:54.600 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQA9WDxmKPPLKxAAGlyhDb222M4GpRybQ24Z3g== 2024-05-09T04:59:54.600 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.5] 2024-05-09T04:59:54.600 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBAWDxmRKm6KRAAcZrQzTb6SQW5CQPyW9f+uQ== 2024-05-09T04:59:54.600 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.6] 2024-05-09T04:59:54.600 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBDWDxmQkRwKBAA5/kHAwsRHh27fnYJuUh16A== 2024-05-09T04:59:54.600 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.7] 2024-05-09T04:59:54.600 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBGWDxmAc98JhAAA/qroXPpF1fC79NGd7WN4w== 2024-05-09T04:59:54.600 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2024-05-09T04:59:54.600 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAtWDxmiEa1CBAAm5ehYW6MVg0B6WUq8TxNbg== 2024-05-09T04:59:54.600 INFO:teuthology.orchestra.run.smithi060.stdout:[client.2] 2024-05-09T04:59:54.600 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAtWDxmtWd4DhAAF1LGmIJOK3X1UoC2pTnOLw== 2024-05-09T04:59:54.600 INFO:teuthology.orchestra.run.smithi060.stdout:[client.1] 2024-05-09T04:59:54.600 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAtWDxmA/enERAABGFsfdUaqliUPbTu5oCWJQ== 2024-05-09T04:59:54.601 INFO:teuthology.orchestra.run.smithi060.stdout:[client.3] 2024-05-09T04:59:54.601 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAtWDxmFWFIFxAAY/B8R3xXgu+A509Ww5Hqjg== 2024-05-09T04:59:54.620 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr.y] 2024-05-09T04:59:54.620 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAsWDxmdBNeJhAAyU0fz/hMt2qp285unqOtlQ== 2024-05-09T04:59:54.620 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.a] 2024-05-09T04:59:54.620 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAsWDxmJBQ1LRAAJfklX7rdjYuv6WIp+8rgMg== 2024-05-09T04:59:54.620 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.c] 2024-05-09T04:59:54.620 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAsWDxmmr9iNBAAyatU9+hjHFV/AFTKpVxMPQ== 2024-05-09T04:59:54.620 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.0] 2024-05-09T04:59:54.620 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAuWDxmOInrKBAAthThp6oxwvbii2eif7krkg== 2024-05-09T04:59:54.620 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.1] 2024-05-09T04:59:54.620 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAxWDxmbDifJxAAhJ5xGS3YfkNkYuJYF7MiIg== 2024-05-09T04:59:54.620 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.2] 2024-05-09T04:59:54.620 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQA0WDxmrlOkJRAAf6eB9x9UcDCilfbWBIMCGQ== 2024-05-09T04:59:54.621 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.3] 2024-05-09T04:59:54.621 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQA3WDxm2ItKJBAAO4FLP3sH/POba/k7DGFXqA== 2024-05-09T04:59:54.621 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr.x] 2024-05-09T04:59:54.621 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAsWDxmuRPmKRAA8uT8sJ8VL1XVxa6c3RTj6g== 2024-05-09T04:59:54.621 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.b] 2024-05-09T04:59:54.621 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAsWDxmUN0fORAAaKZA9GlWlOfJaY/SYAXNlw== 2024-05-09T04:59:54.621 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.d] 2024-05-09T04:59:54.621 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAtWDxms0CIBBAAsHfza+C91qgfGjrczfoGeg== 2024-05-09T04:59:54.621 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.4] 2024-05-09T04:59:54.621 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQA9WDxmKPPLKxAAGlyhDb222M4GpRybQ24Z3g== 2024-05-09T04:59:54.621 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.5] 2024-05-09T04:59:54.621 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQBAWDxmRKm6KRAAcZrQzTb6SQW5CQPyW9f+uQ== 2024-05-09T04:59:54.621 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.6] 2024-05-09T04:59:54.621 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQBDWDxmQkRwKBAA5/kHAwsRHh27fnYJuUh16A== 2024-05-09T04:59:54.621 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.7] 2024-05-09T04:59:54.622 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQBGWDxmAc98JhAAA/qroXPpF1fC79NGd7WN4w== 2024-05-09T04:59:54.622 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-05-09T04:59:54.622 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAtWDxmiEa1CBAAm5ehYW6MVg0B6WUq8TxNbg== 2024-05-09T04:59:54.622 INFO:teuthology.orchestra.run.smithi114.stdout:[client.2] 2024-05-09T04:59:54.622 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAtWDxmtWd4DhAAF1LGmIJOK3X1UoC2pTnOLw== 2024-05-09T04:59:54.622 INFO:teuthology.orchestra.run.smithi114.stdout:[client.1] 2024-05-09T04:59:54.622 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAtWDxmA/enERAABGFsfdUaqliUPbTu5oCWJQ== 2024-05-09T04:59:54.622 INFO:teuthology.orchestra.run.smithi114.stdout:[client.3] 2024-05-09T04:59:54.622 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAtWDxmFWFIFxAAY/B8R3xXgu+A509Ww5Hqjg== 2024-05-09T04:59:54.623 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T04:59:54.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T04:59:54.717 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T04:59:54.720 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T04:59:54.778 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T04:59:54.780 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T04:59:54.835 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T04:59:54.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T04:59:54.888 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T04:59:54.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T04:59:54.941 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T04:59:54.943 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T04:59:54.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T04:59:54.997 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T04:59:55.049 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T04:59:55.051 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T04:59:55.101 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T04:59:55.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T04:59:55.157 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T04:59:55.160 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T04:59:55.213 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T04:59:55.216 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T04:59:55.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T04:59:55.268 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T04:59:55.319 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T04:59:55.321 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T04:59:55.371 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T04:59:55.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T04:59:55.424 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T04:59:55.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T04:59:55.480 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T04:59:55.482 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T04:59:55.535 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T04:59:55.537 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T04:59:55.591 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T04:59:55.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T04:59:55.647 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-09T04:59:55.647 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-05-09T04:59:55.673 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-09T04:59:55.769 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-05-09T04:59:55.795 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-05-09T04:59:55.819 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-09T04:59:55.915 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-05-09T04:59:55.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-05-09T04:59:56.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-09T04:59:56.101 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-05-09T04:59:56.125 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-09T04:59:56.128 DEBUG:teuthology.orchestra.run.smithi114:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-09T04:59:56.180 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-05-09T04:59:56.180 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-09T04:59:56.180 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-05-09T04:59:56.184 INFO:tasks.ceph.mon.a:Started 2024-05-09T04:59:56.185 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-09T04:59:56.185 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-09T04:59:56.222 INFO:tasks.ceph.mon.b:Started 2024-05-09T04:59:56.223 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-09T04:59:56.223 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-09T04:59:56.224 INFO:tasks.ceph.mon.c:Started 2024-05-09T04:59:56.224 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-05-09T04:59:56.225 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-05-09T04:59:56.225 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-05-09T04:59:56.229 INFO:tasks.ceph.mgr.y:Started 2024-05-09T04:59:56.229 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-05-09T04:59:56.229 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-05-09T04:59:56.231 INFO:tasks.ceph.mgr.x:Started 2024-05-09T04:59:56.232 INFO:tasks.ceph:Setting crush tunables to default 2024-05-09T04:59:56.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd crush tunables default 2024-05-09T04:59:56.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.336+0000 7fcb56883640 1 Processor -- start 2024-05-09T04:59:56.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.336+0000 7fcb56883640 1 -- start start 2024-05-09T04:59:56.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.337+0000 7fcb56883640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fcb5013f1f0 0x7fcb5015fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:56.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.337+0000 7fcb56883640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb501418f0 con 0x7fcb5013ee10 2024-05-09T04:59:56.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.337+0000 7fcb56883640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb50141bd0 con 0x7fcb500b2450 2024-05-09T04:59:56.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.337+0000 7fcb56883640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fcb501426b0 con 0x7fcb5013f1f0 2024-05-09T04:59:56.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.337+0000 7fcb4ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcb5013ee10 0x7fcb500b1d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42232/0 (socket says 172.21.15.60:42232) 2024-05-09T04:59:56.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.337+0000 7fcb4ffff640 1 -- 172.21.15.60:0/326605591 learned_addr learned my addr 172.21.15.60:0/326605591 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T04:59:56.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.337+0000 7fcb47fff640 1 --2- 172.21.15.60:0/326605591 >> v2:172.21.15.114:3300/0 conn(0x7fcb5013f1f0 0x7fcb5015fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:56.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.337+0000 7fcb47fff640 1 -- 172.21.15.60:0/326605591 >> v2:172.21.15.114:3300/0 conn(0x7fcb5013f1f0 msgr2=0x7fcb5015fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T04:59:56.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.337+0000 7fcb47fff640 1 -- 172.21.15.60:0/326605591 >> v2:172.21.15.114:3300/0 conn(0x7fcb5013f1f0 msgr2=0x7fcb5015fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T04:59:56.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.337+0000 7fcb47fff640 1 --2- 172.21.15.60:0/326605591 >> v2:172.21.15.114:3300/0 conn(0x7fcb5013f1f0 0x7fcb5015fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T04:59:56.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.337+0000 7fcb47fff640 1 --2- 172.21.15.60:0/326605591 >> v2:172.21.15.114:3300/0 conn(0x7fcb5013f1f0 0x7fcb5015fc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T04:59:56.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.498+0000 7fcb4f7fe640 1 -- 172.21.15.60:0/326605591 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3397972758 0 0) 0x7fcb501418f0 con 0x7fcb5013ee10 2024-05-09T04:59:56.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.498+0000 7fcb4f7fe640 1 -- 172.21.15.60:0/326605591 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb3c003520 con 0x7fcb5013ee10 2024-05-09T04:59:56.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.498+0000 7fcb4f7fe640 1 -- 172.21.15.60:0/326605591 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3892287896 0 0) 0x7fcb50141bd0 con 0x7fcb500b2450 2024-05-09T04:59:56.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.498+0000 7fcb4f7fe640 1 -- 172.21.15.60:0/326605591 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb501418f0 con 0x7fcb500b2450 2024-05-09T04:59:56.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.498+0000 7fcb4f7fe640 1 -- 172.21.15.60:0/326605591 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2185949540 0 0) 0x7fcb3c003520 con 0x7fcb5013ee10 2024-05-09T04:59:56.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.498+0000 7fcb4f7fe640 1 -- 172.21.15.60:0/326605591 >> v2:172.21.15.114:3300/0 conn(0x7fcb5013f1f0 msgr2=0x7fcb5015fc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T04:59:56.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.498+0000 7fcb4f7fe640 1 --2- 172.21.15.60:0/326605591 >> v2:172.21.15.114:3300/0 conn(0x7fcb5013f1f0 0x7fcb5015fc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:56.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.498+0000 7fcb4f7fe640 1 -- 172.21.15.60:0/326605591 >> v1:172.21.15.114:6789/0 conn(0x7fcb500b2450 legacy=0x7fcb50160170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:56.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.498+0000 7fcb4f7fe640 1 -- 172.21.15.60:0/326605591 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb50141eb0 con 0x7fcb5013ee10 2024-05-09T04:59:56.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.498+0000 7fcb4f7fe640 1 -- 172.21.15.60:0/326605591 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7fcb400016f0 con 0x7fcb5013ee10 2024-05-09T04:59:56.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.498+0000 7fcb4f7fe640 1 -- 172.21.15.60:0/326605591 >> v1:172.21.15.60:6789/0 conn(0x7fcb5013ee10 legacy=0x7fcb500b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:56.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.499+0000 7fcb4f7fe640 1 --2- 172.21.15.60:0/326605591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb3c003c30 0x7fcb3c004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:56.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.499+0000 7fcb4f7fe640 1 --2- 172.21.15.60:0/326605591 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb3c004800 0x7fcb3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:56.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.499+0000 7fcb4f7fe640 1 --2- 172.21.15.60:0/326605591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb3c005290 0x7fcb3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:56.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.499+0000 7fcb4f7fe640 1 -- 172.21.15.60:0/326605591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fcb501426b0 con 0x7fcb3c005290 2024-05-09T04:59:56.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.499+0000 7fcb4f7fe640 1 -- 172.21.15.60:0/326605591 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3c00c1e0 con 0x7fcb3c003c30 2024-05-09T04:59:56.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.499+0000 7fcb4f7fe640 1 -- 172.21.15.60:0/326605591 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fcb3c00c3e0 con 0x7fcb3c004800 2024-05-09T04:59:56.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.499+0000 7fcb54e20640 1 --2- 172.21.15.60:0/326605591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb3c005290 0x7fcb3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:56.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.499+0000 7fcb47fff640 1 --2- 172.21.15.60:0/326605591 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb3c004800 0x7fcb3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:56.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.499+0000 7fcb4ffff640 1 --2- 172.21.15.60:0/326605591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb3c003c30 0x7fcb3c004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:56.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.499+0000 7fcb54e20640 1 -- 172.21.15.60:0/326605591 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb3c004800 msgr2=0x7fcb3c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T04:59:56.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.499+0000 7fcb54e20640 1 --2- 172.21.15.60:0/326605591 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb3c004800 0x7fcb3c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:56.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.499+0000 7fcb54e20640 1 -- 172.21.15.60:0/326605591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb3c003c30 msgr2=0x7fcb3c004030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T04:59:56.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.499+0000 7fcb54e20640 1 --2- 172.21.15.60:0/326605591 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb3c003c30 0x7fcb3c004030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:56.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.499+0000 7fcb54e20640 1 -- 172.21.15.60:0/326605591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb3c003520 con 0x7fcb3c005290 2024-05-09T04:59:56.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.500+0000 7fcb54e20640 1 -- 172.21.15.60:0/326605591 reap_dead start 2024-05-09T04:59:56.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.500+0000 7fcb54e20640 1 --2- 172.21.15.60:0/326605591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb3c005290 0x7fcb3c00b9c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcb48001fe0 tx=0x7fcb3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c66ff782a824a2f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:56.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.500+0000 7fcb4f7fe640 1 -- 172.21.15.60:0/326605591 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb48001cb0 con 0x7fcb3c005290 2024-05-09T04:59:56.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.501+0000 7fcb4f7fe640 1 -- 172.21.15.60:0/326605591 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb48003970 con 0x7fcb3c005290 2024-05-09T04:59:56.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.501+0000 7fcb4f7fe640 1 -- 172.21.15.60:0/326605591 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4800e2d0 con 0x7fcb3c005290 2024-05-09T04:59:56.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.501+0000 7fcb56883640 1 -- 172.21.15.60:0/326605591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb3c005290 msgr2=0x7fcb3c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:56.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.501+0000 7fcb56883640 1 --2- 172.21.15.60:0/326605591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb3c005290 0x7fcb3c00b9c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcb48001fe0 tx=0x7fcb3c005830 comp rx=0 tx=0).stop 2024-05-09T04:59:56.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.501+0000 7fcb56883640 1 -- 172.21.15.60:0/326605591 shutdown_connections 2024-05-09T04:59:56.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.501+0000 7fcb56883640 1 --2- 172.21.15.60:0/326605591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb3c005290 0x7fcb3c00b9c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:56.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.501+0000 7fcb56883640 1 -- 172.21.15.60:0/326605591 >> 172.21.15.60:0/326605591 conn(0x7fcb500b4220 msgr2=0x7fcb500b4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:56.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.502+0000 7fcb56883640 1 -- 172.21.15.60:0/326605591 shutdown_connections 2024-05-09T04:59:56.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.502+0000 7fcb56883640 1 -- 172.21.15.60:0/326605591 wait complete. 2024-05-09T04:59:56.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.503+0000 7fcb56883640 1 Processor -- start 2024-05-09T04:59:56.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.503+0000 7fcb56883640 1 -- start start 2024-05-09T04:59:56.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.504+0000 7fcb56883640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb3c005290 0x7fcb5012f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:56.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.504+0000 7fcb56883640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb50152f60 0x7fcb50154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:56.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.504+0000 7fcb56883640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb50155490 0x7fcb5006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:56.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.504+0000 7fcb56883640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fcb500b20d0 con 0x7fcb3c005290 2024-05-09T04:59:56.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.504+0000 7fcb56883640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcb500b1f50 con 0x7fcb50155490 2024-05-09T04:59:56.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.504+0000 7fcb56883640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fcb500b2250 con 0x7fcb50152f60 2024-05-09T04:59:56.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.504+0000 7fcb4ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb3c005290 0x7fcb5012f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:56.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.504+0000 7fcb4ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb3c005290 0x7fcb5012f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43436/0 (socket says 172.21.15.60:43436) 2024-05-09T04:59:56.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.504+0000 7fcb4ffff640 1 -- 172.21.15.60:0/4285056793 learned_addr learned my addr 172.21.15.60:0/4285056793 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T04:59:56.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.504+0000 7fcb47fff640 1 --2- 172.21.15.60:0/4285056793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb50152f60 0x7fcb50154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.504+0000 7fcb54e20640 1 --2- 172.21.15.60:0/4285056793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb50155490 0x7fcb5006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.504+0000 7fcb4ffff640 1 -- 172.21.15.60:0/4285056793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb50152f60 msgr2=0x7fcb50154f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.504+0000 7fcb4ffff640 1 --2- 172.21.15.60:0/4285056793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb50152f60 0x7fcb50154f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.504+0000 7fcb4ffff640 1 -- 172.21.15.60:0/4285056793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb50155490 msgr2=0x7fcb5006e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.504+0000 7fcb4ffff640 1 --2- 172.21.15.60:0/4285056793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb50155490 0x7fcb5006e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.504+0000 7fcb4ffff640 1 -- 172.21.15.60:0/4285056793 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb5012f760 con 0x7fcb3c005290 2024-05-09T04:59:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.504+0000 7fcb54e20640 1 --2- 172.21.15.60:0/4285056793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb50155490 0x7fcb5006e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T04:59:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.504+0000 7fcb4ffff640 1 --2- 172.21.15.60:0/4285056793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb3c005290 0x7fcb5012f220 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcb400072c0 tx=0x7fcb40007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.506+0000 7fcb4d7fa640 1 -- 172.21.15.60:0/4285056793 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb400016d0 con 0x7fcb3c005290 2024-05-09T04:59:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.506+0000 7fcb4d7fa640 1 -- 172.21.15.60:0/4285056793 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb40018070 con 0x7fcb3c005290 2024-05-09T04:59:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.506+0000 7fcb4d7fa640 1 -- 172.21.15.60:0/4285056793 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4000fba0 con 0x7fcb3c005290 2024-05-09T04:59:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.506+0000 7fcb56883640 1 -- 172.21.15.60:0/4285056793 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb50153eb0 con 0x7fcb3c005290 2024-05-09T04:59:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.506+0000 7fcb56883640 1 -- 172.21.15.60:0/4285056793 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb50154470 con 0x7fcb3c005290 2024-05-09T04:59:56.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.506+0000 7fcb4d7fa640 1 -- 172.21.15.60:0/4285056793 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcb4000fd40 con 0x7fcb3c005290 2024-05-09T04:59:56.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.506+0000 7fcb4d7fa640 1 -- 172.21.15.60:0/4285056793 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcb400135f0 con 0x7fcb3c005290 2024-05-09T04:59:56.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.507+0000 7fcb56883640 1 -- 172.21.15.60:0/4285056793 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb50051040 con 0x7fcb3c005290 2024-05-09T04:59:56.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.512+0000 7fcb4d7fa640 1 -- 172.21.15.60:0/4285056793 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fcb400138f0 con 0x7fcb3c005290 2024-05-09T04:59:56.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.579+0000 7fcb56883640 1 -- 172.21.15.60:0/4285056793 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fcb500d0740 con 0x7fcb3c005290 2024-05-09T04:59:56.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.581+0000 7fcb4d7fa640 1 -- 172.21.15.60:0/4285056793 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fcb40013ad0 con 0x7fcb3c005290 2024-05-09T04:59:56.581 INFO:teuthology.orchestra.run.smithi060.stderr:adjusted tunables profile to default 2024-05-09T04:59:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.582+0000 7fcb56883640 1 -- 172.21.15.60:0/4285056793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb3c005290 msgr2=0x7fcb5012f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.582+0000 7fcb56883640 1 --2- 172.21.15.60:0/4285056793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb3c005290 0x7fcb5012f220 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcb400072c0 tx=0x7fcb40007780 comp rx=0 tx=0).stop 2024-05-09T04:59:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.582+0000 7fcb56883640 1 -- 172.21.15.60:0/4285056793 shutdown_connections 2024-05-09T04:59:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.582+0000 7fcb56883640 1 --2- 172.21.15.60:0/4285056793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb50155490 0x7fcb5006e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.582+0000 7fcb56883640 1 --2- 172.21.15.60:0/4285056793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcb50152f60 0x7fcb50154f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.582+0000 7fcb56883640 1 --2- 172.21.15.60:0/4285056793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb3c005290 0x7fcb5012f220 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.582+0000 7fcb56883640 1 -- 172.21.15.60:0/4285056793 >> 172.21.15.60:0/4285056793 conn(0x7fcb500b4220 msgr2=0x7fcb5015a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.582+0000 7fcb56883640 1 -- 172.21.15.60:0/4285056793 shutdown_connections 2024-05-09T04:59:56.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:56.582+0000 7fcb56883640 1 -- 172.21.15.60:0/4285056793 wait complete. 2024-05-09T04:59:56.597 INFO:tasks.ceph:check_enable_crimson: False 2024-05-09T04:59:56.598 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-05-09T04:59:56.598 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:56.598 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-05-09T04:59:56.607 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:56.607+0000 7f74e271c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T04:59:56.621 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T04:59:56.621+0000 7f4fbd7ff040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T04:59:56.624 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:56.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-05-09T04:59:56.690 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:56.690 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-05-09T04:59:56.756 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T04:59:56.756 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-05-09T04:59:56.779 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T04:59:56.779+0000 7f4fbd7ff040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T04:59:56.783 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:56.783+0000 7f74e271c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T04:59:56.804 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:56.804 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-05-09T04:59:56.829 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:56.829 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-05-09T04:59:56.836 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T04:59:56.836+0000 7f4fbd7ff040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T04:59:56.839 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:56.838+0000 7f74e271c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T04:59:56.863 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:56.863 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-05-09T04:59:56.928 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T04:59:56.928 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-05-09T04:59:56.993 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new bccd2765-9913-4f63-8661-ad7f5337d356 0 2024-05-09T04:59:57.096 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T04:59:57.096+0000 7f4fbd7ff040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T04:59:57.099 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:57.098+0000 7f74e271c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T04:59:57.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.131+0000 7fc711fe7640 1 Processor -- start 2024-05-09T04:59:57.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.131+0000 7fc711fe7640 1 -- start start 2024-05-09T04:59:57.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.132+0000 7fc711fe7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc70c0b72f0 0x7fc70c145bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.132+0000 7fc711fe7640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc70c0b6520 con 0x7fc70c146cd0 2024-05-09T04:59:57.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.132+0000 7fc711fe7640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc70c161b90 con 0x7fc70c15f0e0 2024-05-09T04:59:57.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.132+0000 7fc711fe7640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fc70c0b5f60 con 0x7fc70c0b72f0 2024-05-09T04:59:57.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.132+0000 7fc70bfff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fc70c15f0e0 0x7fc70c161480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48352/0 (socket says 172.21.15.114:48352) 2024-05-09T04:59:57.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.132+0000 7fc70bfff640 1 -- 172.21.15.114:0/2246955762 learned_addr learned my addr 172.21.15.114:0/2246955762 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-05-09T04:59:57.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.133+0000 7fc70b7fe640 1 --2- 172.21.15.114:0/2246955762 >> v2:172.21.15.60:3300/0 conn(0x7fc70c0b72f0 0x7fc70c145bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.133+0000 7fc70a7fc640 1 -- 172.21.15.114:0/2246955762 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 876803368 0 0) 0x7fc70c161b90 con 0x7fc70c15f0e0 2024-05-09T04:59:57.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.133+0000 7fc70a7fc640 1 -- 172.21.15.114:0/2246955762 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6f8003520 con 0x7fc70c15f0e0 2024-05-09T04:59:57.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.133+0000 7fc70a7fc640 1 -- 172.21.15.114:0/2246955762 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2882773866 0 0) 0x7fc70c0b6520 con 0x7fc70c146cd0 2024-05-09T04:59:57.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.133+0000 7fc70a7fc640 1 -- 172.21.15.114:0/2246955762 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc70c161b90 con 0x7fc70c146cd0 2024-05-09T04:59:57.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.133+0000 7fc70b7fe640 1 -- 172.21.15.114:0/2246955762 >> v1:172.21.15.114:6789/0 conn(0x7fc70c15f0e0 legacy=0x7fc70c161480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.133+0000 7fc70b7fe640 1 -- 172.21.15.114:0/2246955762 >> v1:172.21.15.60:6789/0 conn(0x7fc70c146cd0 legacy=0x7fc70c160d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.133+0000 7fc70b7fe640 1 -- 172.21.15.114:0/2246955762 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc70c0b6240 con 0x7fc70c0b72f0 2024-05-09T04:59:57.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.134+0000 7fc70b7fe640 1 --2- 172.21.15.114:0/2246955762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc70c0b72f0 0x7fc70c145bd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40097a0 tx=0x7fc6f402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5d5ea5ee6c9e5f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:57.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.134+0000 7fc70a7fc640 1 -- 172.21.15.114:0/2246955762 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4037070 con 0x7fc70c0b72f0 2024-05-09T04:59:57.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.134+0000 7fc70a7fc640 1 -- 172.21.15.114:0/2246955762 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6f4032440 con 0x7fc70c0b72f0 2024-05-09T04:59:57.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.134+0000 7fc70a7fc640 1 -- 172.21.15.114:0/2246955762 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4032810 con 0x7fc70c0b72f0 2024-05-09T04:59:57.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.134+0000 7fc711fe7640 1 -- 172.21.15.114:0/2246955762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc70c0b72f0 msgr2=0x7fc70c145bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.134+0000 7fc711fe7640 1 --2- 172.21.15.114:0/2246955762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc70c0b72f0 0x7fc70c145bd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40097a0 tx=0x7fc6f402a230 comp rx=0 tx=0).stop 2024-05-09T04:59:57.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.135+0000 7fc711fe7640 1 -- 172.21.15.114:0/2246955762 shutdown_connections 2024-05-09T04:59:57.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.135+0000 7fc711fe7640 1 --2- 172.21.15.114:0/2246955762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc70c0b72f0 0x7fc70c145bd0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.135+0000 7fc711fe7640 1 -- 172.21.15.114:0/2246955762 >> 172.21.15.114:0/2246955762 conn(0x7fc70c0b3320 msgr2=0x7fc70c070590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:57.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.135+0000 7fc711fe7640 1 -- 172.21.15.114:0/2246955762 shutdown_connections 2024-05-09T04:59:57.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.135+0000 7fc711fe7640 1 -- 172.21.15.114:0/2246955762 wait complete. 2024-05-09T04:59:57.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.136+0000 7fc711fe7640 1 Processor -- start 2024-05-09T04:59:57.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.136+0000 7fc711fe7640 1 -- start start 2024-05-09T04:59:57.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.137+0000 7fc711fe7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc70c0b72f0 0x7fc70c15cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.137+0000 7fc711fe7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc70c146cd0 0x7fc70c13cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.137+0000 7fc711fe7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc70c15f0e0 0x7fc70c139ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.137+0000 7fc711fe7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc70c0d2570 con 0x7fc70c0b72f0 2024-05-09T04:59:57.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.137+0000 7fc711fe7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc70c15dbb0 con 0x7fc70c15f0e0 2024-05-09T04:59:57.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.137+0000 7fc711fe7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc70c0d26f0 con 0x7fc70c146cd0 2024-05-09T04:59:57.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.137+0000 7fc70affd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc70c146cd0 0x7fc70c13cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.137+0000 7fc70bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc70c15f0e0 0x7fc70c139ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.137+0000 7fc70b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc70c0b72f0 0x7fc70c15cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.137+0000 7fc70affd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc70c146cd0 0x7fc70c13cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:39910/0 (socket says 172.21.15.114:39910) 2024-05-09T04:59:57.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.137+0000 7fc70affd640 1 -- 172.21.15.114:0/2131991184 learned_addr learned my addr 172.21.15.114:0/2131991184 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T04:59:57.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.137+0000 7fc70b7fe640 1 -- 172.21.15.114:0/2131991184 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc70c146cd0 msgr2=0x7fc70c13cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.137+0000 7fc70b7fe640 1 --2- 172.21.15.114:0/2131991184 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc70c146cd0 0x7fc70c13cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.137+0000 7fc70b7fe640 1 -- 172.21.15.114:0/2131991184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc70c15f0e0 msgr2=0x7fc70c139ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.137+0000 7fc70b7fe640 1 --2- 172.21.15.114:0/2131991184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc70c15f0e0 0x7fc70c139ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.137+0000 7fc70b7fe640 1 -- 172.21.15.114:0/2131991184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc70c13d4a0 con 0x7fc70c0b72f0 2024-05-09T04:59:57.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.137+0000 7fc70affd640 1 --2- 172.21.15.114:0/2131991184 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc70c146cd0 0x7fc70c13cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T04:59:57.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.137+0000 7fc70bfff640 1 --2- 172.21.15.114:0/2131991184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc70c15f0e0 0x7fc70c139ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T04:59:57.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.137+0000 7fc70b7fe640 1 --2- 172.21.15.114:0/2131991184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc70c0b72f0 0x7fc70c15cb60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0b4d80 tx=0x7fc6f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:57.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.138+0000 7fc703fff640 1 -- 172.21.15.114:0/2131991184 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4041030 con 0x7fc70c0b72f0 2024-05-09T04:59:57.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.138+0000 7fc703fff640 1 -- 172.21.15.114:0/2131991184 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6f403d420 con 0x7fc70c0b72f0 2024-05-09T04:59:57.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.138+0000 7fc711fe7640 1 -- 172.21.15.114:0/2131991184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc70c13d780 con 0x7fc70c0b72f0 2024-05-09T04:59:57.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.138+0000 7fc703fff640 1 -- 172.21.15.114:0/2131991184 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4037040 con 0x7fc70c0b72f0 2024-05-09T04:59:57.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.138+0000 7fc711fe7640 1 -- 172.21.15.114:0/2131991184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc70c13dd40 con 0x7fc70c0b72f0 2024-05-09T04:59:57.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.138+0000 7fc703fff640 1 -- 172.21.15.114:0/2131991184 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc6f4044070 con 0x7fc70c0b72f0 2024-05-09T04:59:57.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.138+0000 7fc703fff640 1 -- 172.21.15.114:0/2131991184 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fc6f4046980 con 0x7fc70c0b72f0 2024-05-09T04:59:57.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.139+0000 7fc711fe7640 1 -- 172.21.15.114:0/2131991184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc70c051040 con 0x7fc70c0b72f0 2024-05-09T04:59:57.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.141+0000 7fc703fff640 1 -- 172.21.15.114:0/2131991184 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fc6f4046bd0 con 0x7fc70c0b72f0 2024-05-09T04:59:57.144 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T04:59:57.144+0000 7f4fbd7ff040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T04:59:57.147 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:57.147+0000 7f74e271c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T04:59:57.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.212+0000 7fc711fe7640 1 -- 172.21.15.114:0/2131991184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bccd2765-9913-4f63-8661-ad7f5337d356", "id": 0} v 0) -- 0x7fc70c0b6e60 con 0x7fc70c0b72f0 2024-05-09T04:59:57.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.214+0000 7fc703fff640 1 -- 172.21.15.114:0/2131991184 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bccd2765-9913-4f63-8661-ad7f5337d356", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fc6f4046db0 con 0x7fc70c0b72f0 2024-05-09T04:59:57.214 INFO:teuthology.orchestra.run.smithi114.stdout:0 2024-05-09T04:59:57.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.215+0000 7fc711fe7640 1 -- 172.21.15.114:0/2131991184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc70c0b72f0 msgr2=0x7fc70c15cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.215+0000 7fc711fe7640 1 --2- 172.21.15.114:0/2131991184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc70c0b72f0 0x7fc70c15cb60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0b4d80 tx=0x7fc6f40046c0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.215+0000 7fc711fe7640 1 -- 172.21.15.114:0/2131991184 shutdown_connections 2024-05-09T04:59:57.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.215+0000 7fc711fe7640 1 --2- 172.21.15.114:0/2131991184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc70c15f0e0 0x7fc70c139ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.215+0000 7fc711fe7640 1 --2- 172.21.15.114:0/2131991184 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc70c146cd0 0x7fc70c13cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.215+0000 7fc711fe7640 1 --2- 172.21.15.114:0/2131991184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc70c0b72f0 0x7fc70c15cb60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.215+0000 7fc711fe7640 1 -- 172.21.15.114:0/2131991184 >> 172.21.15.114:0/2131991184 conn(0x7fc70c0b3320 msgr2=0x7fc70c146570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:57.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.215+0000 7fc711fe7640 1 -- 172.21.15.114:0/2131991184 shutdown_connections 2024-05-09T04:59:57.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.215+0000 7fc711fe7640 1 -- 172.21.15.114:0/2131991184 wait complete. 2024-05-09T04:59:57.226 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new a32ec935-149d-4a8e-a59e-3417d4a61475 1 2024-05-09T04:59:57.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.324+0000 7f5aee0e8640 1 Processor -- start 2024-05-09T04:59:57.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.324+0000 7f5aee0e8640 1 -- start start 2024-05-09T04:59:57.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.324+0000 7f5aee0e8640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5ae8148da0 0x7f5ae815fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.324+0000 7f5aee0e8640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ae8160480 con 0x7f5ae815ffb0 2024-05-09T04:59:57.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.324+0000 7f5aee0e8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ae8160760 con 0x7f5ae81489c0 2024-05-09T04:59:57.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.324+0000 7f5aee0e8640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f5ae8163200 con 0x7f5ae8148da0 2024-05-09T04:59:57.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.325+0000 7f5ae6ffd640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5ae8148da0 0x7f5ae815fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.325+0000 7f5ae77fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5ae81489c0 0x7f5ae8163a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48368/0 (socket says 172.21.15.114:48368) 2024-05-09T04:59:57.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.325+0000 7f5ae77fe640 1 -- 172.21.15.114:0/3550246429 learned_addr learned my addr 172.21.15.114:0/3550246429 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-05-09T04:59:57.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.325+0000 7f5ae6ffd640 1 -- 172.21.15.114:0/3550246429 >> v1:172.21.15.114:6789/0 conn(0x7f5ae81489c0 legacy=0x7f5ae8163a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.325+0000 7f5ae6ffd640 1 -- 172.21.15.114:0/3550246429 >> v1:172.21.15.60:6789/0 conn(0x7f5ae815ffb0 legacy=0x7f5ae8072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.325+0000 7f5ae6ffd640 1 -- 172.21.15.114:0/3550246429 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae81634e0 con 0x7f5ae8148da0 2024-05-09T04:59:57.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.326+0000 7f5ae6ffd640 1 --2- 172.21.15.114:0/3550246429 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae8148da0 0x7f5ae815fa70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5ad80097a0 tx=0x7f5ad802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3989d0a4a79b7d04 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:57.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.326+0000 7f5ae67fc640 1 -- 172.21.15.114:0/3550246429 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad8037070 con 0x7f5ae8148da0 2024-05-09T04:59:57.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.326+0000 7f5ae67fc640 1 -- 172.21.15.114:0/3550246429 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ad8032440 con 0x7f5ae8148da0 2024-05-09T04:59:57.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.326+0000 7f5ae67fc640 1 -- 172.21.15.114:0/3550246429 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad8032810 con 0x7f5ae8148da0 2024-05-09T04:59:57.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.326+0000 7f5aee0e8640 1 -- 172.21.15.114:0/3550246429 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae8148da0 msgr2=0x7f5ae815fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.326+0000 7f5aee0e8640 1 --2- 172.21.15.114:0/3550246429 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae8148da0 0x7f5ae815fa70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5ad80097a0 tx=0x7f5ad802a230 comp rx=0 tx=0).stop 2024-05-09T04:59:57.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.326+0000 7f5aee0e8640 1 -- 172.21.15.114:0/3550246429 shutdown_connections 2024-05-09T04:59:57.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.326+0000 7f5aee0e8640 1 --2- 172.21.15.114:0/3550246429 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae8148da0 0x7f5ae815fa70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.326+0000 7f5aee0e8640 1 -- 172.21.15.114:0/3550246429 >> 172.21.15.114:0/3550246429 conn(0x7f5ae80d20a0 msgr2=0x7f5ae80d2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:57.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.327+0000 7f5aee0e8640 1 -- 172.21.15.114:0/3550246429 shutdown_connections 2024-05-09T04:59:57.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.327+0000 7f5aee0e8640 1 -- 172.21.15.114:0/3550246429 wait complete. 2024-05-09T04:59:57.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.328+0000 7f5aee0e8640 1 Processor -- start 2024-05-09T04:59:57.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.328+0000 7f5aee0e8640 1 -- start start 2024-05-09T04:59:57.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5aee0e8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ae81489c0 0x7f5ae8159620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5aee0e8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae8148da0 0x7f5ae8157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5aee0e8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae815ffb0 0x7f5ae81455c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5aee0e8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae8072840 con 0x7f5ae81489c0 2024-05-09T04:59:57.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5aee0e8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae80b7d10 con 0x7f5ae815ffb0 2024-05-09T04:59:57.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5aee0e8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f5ae80729c0 con 0x7f5ae8148da0 2024-05-09T04:59:57.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5ae6ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae8148da0 0x7f5ae8157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5ae7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae815ffb0 0x7f5ae81455c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5ae6ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae8148da0 0x7f5ae8157560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:39922/0 (socket says 172.21.15.114:39922) 2024-05-09T04:59:57.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5ae6ffd640 1 -- 172.21.15.114:0/3317116235 learned_addr learned my addr 172.21.15.114:0/3317116235 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T04:59:57.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5ae77fe640 1 --2- 172.21.15.114:0/3317116235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ae81489c0 0x7f5ae8159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5ae7fff640 1 -- 172.21.15.114:0/3317116235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae8148da0 msgr2=0x7f5ae8157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5ae7fff640 1 --2- 172.21.15.114:0/3317116235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae8148da0 0x7f5ae8157560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5ae7fff640 1 -- 172.21.15.114:0/3317116235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ae81489c0 msgr2=0x7f5ae8159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5ae7fff640 1 --2- 172.21.15.114:0/3317116235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ae81489c0 0x7f5ae8159620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5ae7fff640 1 -- 172.21.15.114:0/3317116235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae8143a80 con 0x7f5ae815ffb0 2024-05-09T04:59:57.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5ae6ffd640 1 --2- 172.21.15.114:0/3317116235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae8148da0 0x7f5ae8157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T04:59:57.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5ae7fff640 1 --2- 172.21.15.114:0/3317116235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae815ffb0 0x7f5ae81455c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5ae000c730 tx=0x7f5ae000cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:57.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5acffff640 1 -- 172.21.15.114:0/3317116235 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0003b70 con 0x7f5ae815ffb0 2024-05-09T04:59:57.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5acffff640 1 -- 172.21.15.114:0/3317116235 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae0003d10 con 0x7f5ae815ffb0 2024-05-09T04:59:57.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5aee0e8640 1 -- 172.21.15.114:0/3317116235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ae8158580 con 0x7f5ae815ffb0 2024-05-09T04:59:57.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5acffff640 1 -- 172.21.15.114:0/3317116235 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae00052f0 con 0x7f5ae815ffb0 2024-05-09T04:59:57.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.329+0000 7f5aee0e8640 1 -- 172.21.15.114:0/3317116235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ae8158b40 con 0x7f5ae815ffb0 2024-05-09T04:59:57.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.330+0000 7f5acffff640 1 -- 172.21.15.114:0/3317116235 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5ae00056a0 con 0x7f5ae815ffb0 2024-05-09T04:59:57.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.330+0000 7f5acffff640 1 -- 172.21.15.114:0/3317116235 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f5ae00126d0 con 0x7f5ae815ffb0 2024-05-09T04:59:57.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.330+0000 7f5aee0e8640 1 -- 172.21.15.114:0/3317116235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ae8051040 con 0x7f5ae815ffb0 2024-05-09T04:59:57.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.332+0000 7f5acffff640 1 -- 172.21.15.114:0/3317116235 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f5ae0012920 con 0x7f5ae815ffb0 2024-05-09T04:59:57.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.403+0000 7f5aee0e8640 1 -- 172.21.15.114:0/3317116235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a32ec935-149d-4a8e-a59e-3417d4a61475", "id": 1} v 0) -- 0x7f5ae80b6e10 con 0x7f5ae815ffb0 2024-05-09T04:59:57.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.406+0000 7f5acffff640 1 -- 172.21.15.114:0/3317116235 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a32ec935-149d-4a8e-a59e-3417d4a61475", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f5ae0012bd0 con 0x7f5ae815ffb0 2024-05-09T04:59:57.406 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-05-09T04:59:57.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.407+0000 7f5aee0e8640 1 -- 172.21.15.114:0/3317116235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae815ffb0 msgr2=0x7f5ae81455c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.407+0000 7f5aee0e8640 1 --2- 172.21.15.114:0/3317116235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae815ffb0 0x7f5ae81455c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5ae000c730 tx=0x7f5ae000cbf0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.407+0000 7f5aee0e8640 1 -- 172.21.15.114:0/3317116235 shutdown_connections 2024-05-09T04:59:57.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.407+0000 7f5aee0e8640 1 --2- 172.21.15.114:0/3317116235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae815ffb0 0x7f5ae81455c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.407+0000 7f5aee0e8640 1 --2- 172.21.15.114:0/3317116235 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae8148da0 0x7f5ae8157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.407+0000 7f5aee0e8640 1 --2- 172.21.15.114:0/3317116235 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ae81489c0 0x7f5ae8159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.407+0000 7f5aee0e8640 1 -- 172.21.15.114:0/3317116235 >> 172.21.15.114:0/3317116235 conn(0x7f5ae80d20a0 msgr2=0x7f5ae8072110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:57.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.407+0000 7f5aee0e8640 1 -- 172.21.15.114:0/3317116235 shutdown_connections 2024-05-09T04:59:57.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.407+0000 7f5aee0e8640 1 -- 172.21.15.114:0/3317116235 wait complete. 2024-05-09T04:59:57.418 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 0277ac75-0fad-42ea-b2ad-07c34457aa03 2 2024-05-09T04:59:57.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.518+0000 7f4693e99640 1 Processor -- start 2024-05-09T04:59:57.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.518+0000 7f4693e99640 1 -- start start 2024-05-09T04:59:57.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.519+0000 7f4693e99640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f468c1457f0 0x7f468c162fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.519+0000 7f4693e99640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f468c145bd0 0x7f468c0b7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.519+0000 7f4693e99640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f468c0cc750 con 0x7f468c0b77c0 2024-05-09T04:59:57.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.519+0000 7f4693e99640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f468c0b7d40 con 0x7f468c145bd0 2024-05-09T04:59:57.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.519+0000 7f4693e99640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f468c0cca30 con 0x7f468c1457f0 2024-05-09T04:59:57.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.519+0000 7f4691c35640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f468c1457f0 0x7f468c162fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.519+0000 7f4691c35640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f468c1457f0 0x7f468c162fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:39928/0 (socket says 172.21.15.114:39928) 2024-05-09T04:59:57.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.519+0000 7f4691c35640 1 -- 172.21.15.114:0/4276263910 learned_addr learned my addr 172.21.15.114:0/4276263910 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T04:59:57.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.519+0000 7f4691434640 1 --2- 172.21.15.114:0/4276263910 >> v2:172.21.15.60:3300/0 conn(0x7f468c145bd0 0x7f468c0b7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.519+0000 7f4691c35640 1 -- 172.21.15.114:0/4276263910 >> v2:172.21.15.60:3300/0 conn(0x7f468c145bd0 msgr2=0x7f468c0b7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T04:59:57.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.519+0000 7f4691c35640 1 --2- 172.21.15.114:0/4276263910 >> v2:172.21.15.60:3300/0 conn(0x7f468c145bd0 0x7f468c0b7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.519+0000 7f4691c35640 1 -- 172.21.15.114:0/4276263910 >> v1:172.21.15.60:6789/0 conn(0x7f468c0b77c0 legacy=0x7f468c072110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.519+0000 7f4691c35640 1 -- 172.21.15.114:0/4276263910 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f468c0ccd10 con 0x7f468c1457f0 2024-05-09T04:59:57.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.520+0000 7f4691c35640 1 --2- 172.21.15.114:0/4276263910 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f468c1457f0 0x7f468c162fe0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f467c009870 tx=0x7f467c02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f362679c357ad487 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:57.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.520+0000 7f4690c33640 1 -- 172.21.15.114:0/4276263910 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467c037070 con 0x7f468c1457f0 2024-05-09T04:59:57.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.520+0000 7f4690c33640 1 -- 172.21.15.114:0/4276263910 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f467c032440 con 0x7f468c1457f0 2024-05-09T04:59:57.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.520+0000 7f4690c33640 1 -- 172.21.15.114:0/4276263910 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467c032810 con 0x7f468c1457f0 2024-05-09T04:59:57.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.521+0000 7f4693e99640 1 -- 172.21.15.114:0/4276263910 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f468c1457f0 msgr2=0x7f468c162fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.521+0000 7f4693e99640 1 --2- 172.21.15.114:0/4276263910 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f468c1457f0 0x7f468c162fe0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f467c009870 tx=0x7f467c02a270 comp rx=0 tx=0).stop 2024-05-09T04:59:57.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.521+0000 7f4693e99640 1 -- 172.21.15.114:0/4276263910 shutdown_connections 2024-05-09T04:59:57.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.521+0000 7f4693e99640 1 --2- 172.21.15.114:0/4276263910 >> v2:172.21.15.60:3300/0 conn(0x7f468c145bd0 0x7f468c0b7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.521+0000 7f4693e99640 1 --2- 172.21.15.114:0/4276263910 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f468c1457f0 0x7f468c162fe0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.521+0000 7f4693e99640 1 -- 172.21.15.114:0/4276263910 >> 172.21.15.114:0/4276263910 conn(0x7f468c0d20a0 msgr2=0x7f468c0d2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:57.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.521+0000 7f4693e99640 1 -- 172.21.15.114:0/4276263910 shutdown_connections 2024-05-09T04:59:57.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.521+0000 7f4693e99640 1 -- 172.21.15.114:0/4276263910 wait complete. 2024-05-09T04:59:57.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.522+0000 7f4693e99640 1 Processor -- start 2024-05-09T04:59:57.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.522+0000 7f4693e99640 1 -- start start 2024-05-09T04:59:57.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.523+0000 7f4693e99640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f468c0b77c0 0x7f468c15ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.523+0000 7f4693e99640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f468c1457f0 0x7f468c160420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.523+0000 7f4693e99640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f468c145bd0 0x7f468c147540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.523+0000 7f4693e99640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f468c072820 con 0x7f468c0b77c0 2024-05-09T04:59:57.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.523+0000 7f4693e99640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f468c148e40 con 0x7f468c1457f0 2024-05-09T04:59:57.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.523+0000 7f4693e99640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f468c0729a0 con 0x7f468c145bd0 2024-05-09T04:59:57.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.523+0000 7f4692436640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f468c145bd0 0x7f468c147540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.523+0000 7f4691434640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f468c1457f0 0x7f468c160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.523+0000 7f4692436640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f468c145bd0 0x7f468c147540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:39938/0 (socket says 172.21.15.114:39938) 2024-05-09T04:59:57.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.523+0000 7f4692436640 1 -- 172.21.15.114:0/514808973 learned_addr learned my addr 172.21.15.114:0/514808973 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T04:59:57.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.523+0000 7f4691c35640 1 --2- 172.21.15.114:0/514808973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f468c0b77c0 0x7f468c15ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.523+0000 7f4692436640 1 -- 172.21.15.114:0/514808973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f468c1457f0 msgr2=0x7f468c160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.523+0000 7f4692436640 1 --2- 172.21.15.114:0/514808973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f468c1457f0 0x7f468c160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.523+0000 7f4692436640 1 -- 172.21.15.114:0/514808973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f468c0b77c0 msgr2=0x7f468c15ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.523+0000 7f4692436640 1 --2- 172.21.15.114:0/514808973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f468c0b77c0 0x7f468c15ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.523+0000 7f4692436640 1 -- 172.21.15.114:0/514808973 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f468c160dc0 con 0x7f468c145bd0 2024-05-09T04:59:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.524+0000 7f4692436640 1 --2- 172.21.15.114:0/514808973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f468c145bd0 0x7f468c147540 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f468400c7d0 tx=0x7f468400cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.524+0000 7f46827fc640 1 -- 172.21.15.114:0/514808973 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684003b70 con 0x7f468c145bd0 2024-05-09T04:59:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.524+0000 7f46827fc640 1 -- 172.21.15.114:0/514808973 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4684003d10 con 0x7f468c145bd0 2024-05-09T04:59:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.524+0000 7f4693e99640 1 -- 172.21.15.114:0/514808973 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f468c1610a0 con 0x7f468c145bd0 2024-05-09T04:59:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.524+0000 7f46827fc640 1 -- 172.21.15.114:0/514808973 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468400aa70 con 0x7f468c145bd0 2024-05-09T04:59:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.524+0000 7f4693e99640 1 -- 172.21.15.114:0/514808973 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f468c161660 con 0x7f468c145bd0 2024-05-09T04:59:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.524+0000 7f46827fc640 1 -- 172.21.15.114:0/514808973 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f468400ac10 con 0x7f468c145bd0 2024-05-09T04:59:57.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.524+0000 7f46827fc640 1 -- 172.21.15.114:0/514808973 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f4684012980 con 0x7f468c145bd0 2024-05-09T04:59:57.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.525+0000 7f4693e99640 1 -- 172.21.15.114:0/514808973 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f468c051040 con 0x7f468c145bd0 2024-05-09T04:59:57.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.527+0000 7f46827fc640 1 -- 172.21.15.114:0/514808973 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f4684012c20 con 0x7f468c145bd0 2024-05-09T04:59:57.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.598+0000 7f4693e99640 1 -- 172.21.15.114:0/514808973 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0277ac75-0fad-42ea-b2ad-07c34457aa03", "id": 2} v 0) -- 0x7f468c162d50 con 0x7f468c145bd0 2024-05-09T04:59:57.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.601+0000 7f46827fc640 1 -- 172.21.15.114:0/514808973 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0277ac75-0fad-42ea-b2ad-07c34457aa03", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f4684012560 con 0x7f468c145bd0 2024-05-09T04:59:57.602 INFO:teuthology.orchestra.run.smithi114.stdout:2 2024-05-09T04:59:57.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.602+0000 7f4693e99640 1 -- 172.21.15.114:0/514808973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f468c145bd0 msgr2=0x7f468c147540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.602+0000 7f4693e99640 1 --2- 172.21.15.114:0/514808973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f468c145bd0 0x7f468c147540 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f468400c7d0 tx=0x7f468400cc90 comp rx=0 tx=0).stop 2024-05-09T04:59:57.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.602+0000 7f4693e99640 1 -- 172.21.15.114:0/514808973 shutdown_connections 2024-05-09T04:59:57.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.602+0000 7f4693e99640 1 --2- 172.21.15.114:0/514808973 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f468c145bd0 0x7f468c147540 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.602+0000 7f4693e99640 1 --2- 172.21.15.114:0/514808973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f468c1457f0 0x7f468c160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.602+0000 7f4693e99640 1 --2- 172.21.15.114:0/514808973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f468c0b77c0 0x7f468c15ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.602+0000 7f4693e99640 1 -- 172.21.15.114:0/514808973 >> 172.21.15.114:0/514808973 conn(0x7f468c0d20a0 msgr2=0x7f468c06c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:57.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.602+0000 7f4693e99640 1 -- 172.21.15.114:0/514808973 shutdown_connections 2024-05-09T04:59:57.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.602+0000 7f4693e99640 1 -- 172.21.15.114:0/514808973 wait complete. 2024-05-09T04:59:57.613 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 0978499e-d158-435a-8623-18c8133c31ed 3 2024-05-09T04:59:57.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.712+0000 7f41fd538640 1 Processor -- start 2024-05-09T04:59:57.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.712+0000 7f41fd538640 1 -- start start 2024-05-09T04:59:57.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.713+0000 7f41fd538640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41f8160160 0x7f41f8160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.713+0000 7f41fd538640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f41f80b84e0 0x7f41f80ba230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.713+0000 7f41fd538640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41f80b8ef0 con 0x7f41f80b8920 2024-05-09T04:59:57.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.713+0000 7f41fd538640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f41f80b9790 con 0x7f41f80b84e0 2024-05-09T04:59:57.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.713+0000 7f41fd538640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f41f80b91d0 con 0x7f41f8160160 2024-05-09T04:59:57.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.713+0000 7f41f67fc640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f41f80b84e0 0x7f41f80ba230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.713+0000 7f41f6ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41f8160160 0x7f41f8160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.713+0000 7f41f77fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f41f80b8920 0x7f41f8162fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48380/0 (socket says 172.21.15.114:48380) 2024-05-09T04:59:57.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.713+0000 7f41f77fe640 1 -- 172.21.15.114:0/1076607004 learned_addr learned my addr 172.21.15.114:0/1076607004 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-05-09T04:59:57.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.713+0000 7f41f67fc640 1 -- 172.21.15.114:0/1076607004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41f8160160 msgr2=0x7f41f8160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T04:59:57.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.713+0000 7f41f67fc640 1 --2- 172.21.15.114:0/1076607004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41f8160160 0x7f41f8160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.713+0000 7f41f67fc640 1 -- 172.21.15.114:0/1076607004 >> v1:172.21.15.114:6789/0 conn(0x7f41f80b8920 legacy=0x7f41f8162fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.713+0000 7f41f5ffb640 1 -- 172.21.15.114:0/1076607004 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3716380183 0 0) 0x7f41f80b8ef0 con 0x7f41f80b8920 2024-05-09T04:59:57.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.713+0000 7f41f67fc640 1 -- 172.21.15.114:0/1076607004 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f80b94b0 con 0x7f41f80b84e0 2024-05-09T04:59:57.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.713+0000 7f41f6ffd640 1 --2- 172.21.15.114:0/1076607004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41f8160160 0x7f41f8160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T04:59:57.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.714+0000 7f41f67fc640 1 --2- 172.21.15.114:0/1076607004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41f80b84e0 0x7f41f80ba230 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f41e00097a0 tx=0x7f41e002a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efdbd24d522c3b13 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:57.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.714+0000 7f41f5ffb640 1 -- 172.21.15.114:0/1076607004 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e0037070 con 0x7f41f80b84e0 2024-05-09T04:59:57.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.714+0000 7f41f5ffb640 1 -- 172.21.15.114:0/1076607004 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41e002ae50 con 0x7f41f80b84e0 2024-05-09T04:59:57.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.714+0000 7f41f5ffb640 1 -- 172.21.15.114:0/1076607004 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e0032660 con 0x7f41f80b84e0 2024-05-09T04:59:57.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.715+0000 7f41fd538640 1 -- 172.21.15.114:0/1076607004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41f80b84e0 msgr2=0x7f41f80ba230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.715+0000 7f41fd538640 1 --2- 172.21.15.114:0/1076607004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41f80b84e0 0x7f41f80ba230 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f41e00097a0 tx=0x7f41e002a100 comp rx=0 tx=0).stop 2024-05-09T04:59:57.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.715+0000 7f41fd538640 1 -- 172.21.15.114:0/1076607004 shutdown_connections 2024-05-09T04:59:57.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.715+0000 7f41fd538640 1 --2- 172.21.15.114:0/1076607004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41f80b84e0 0x7f41f80ba230 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.715+0000 7f41fd538640 1 --2- 172.21.15.114:0/1076607004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41f8160160 0x7f41f8160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.715+0000 7f41fd538640 1 -- 172.21.15.114:0/1076607004 >> 172.21.15.114:0/1076607004 conn(0x7f41f806eda0 msgr2=0x7f41f8073820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:57.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.715+0000 7f41fd538640 1 -- 172.21.15.114:0/1076607004 shutdown_connections 2024-05-09T04:59:57.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.715+0000 7f41fd538640 1 -- 172.21.15.114:0/1076607004 wait complete. 2024-05-09T04:59:57.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.716+0000 7f41fd538640 1 Processor -- start 2024-05-09T04:59:57.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.716+0000 7f41fd538640 1 -- start start 2024-05-09T04:59:57.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.717+0000 7f41fd538640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41f80b84e0 0x7f41f8157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.717+0000 7f41fd538640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41f80b8920 0x7f41f8154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.717+0000 7f41fd538640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41f8160160 0x7f41f8160540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.717+0000 7f41fd538640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f41f815fd90 con 0x7f41f80b8920 2024-05-09T04:59:57.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.717+0000 7f41fd538640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f41f815fc10 con 0x7f41f80b84e0 2024-05-09T04:59:57.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.717+0000 7f41fd538640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f41f80b6b10 con 0x7f41f8160160 2024-05-09T04:59:57.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.717+0000 7f41f6ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41f80b84e0 0x7f41f8157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.717+0000 7f41f6ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41f80b84e0 0x7f41f8157140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52212/0 (socket says 172.21.15.114:52212) 2024-05-09T04:59:57.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.717+0000 7f41f6ffd640 1 -- 172.21.15.114:0/2865605459 learned_addr learned my addr 172.21.15.114:0/2865605459 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T04:59:57.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.717+0000 7f41f77fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41f8160160 0x7f41f8160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.717+0000 7f41f67fc640 1 --2- 172.21.15.114:0/2865605459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41f80b8920 0x7f41f8154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.717+0000 7f41f6ffd640 1 -- 172.21.15.114:0/2865605459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41f8160160 msgr2=0x7f41f8160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.717+0000 7f41f6ffd640 1 --2- 172.21.15.114:0/2865605459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41f8160160 0x7f41f8160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.717+0000 7f41f6ffd640 1 -- 172.21.15.114:0/2865605459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41f80b8920 msgr2=0x7f41f8154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.717+0000 7f41f6ffd640 1 --2- 172.21.15.114:0/2865605459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41f80b8920 0x7f41f8154640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.717+0000 7f41f6ffd640 1 -- 172.21.15.114:0/2865605459 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f81550c0 con 0x7f41f80b84e0 2024-05-09T04:59:57.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.717+0000 7f41f67fc640 1 --2- 172.21.15.114:0/2865605459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41f80b8920 0x7f41f8154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T04:59:57.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.717+0000 7f41f6ffd640 1 --2- 172.21.15.114:0/2865605459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41f80b84e0 0x7f41f8157140 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f41e800ebe0 tx=0x7f41e800c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:57.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.718+0000 7f41ef7fe640 1 -- 172.21.15.114:0/2865605459 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e8017070 con 0x7f41f80b84e0 2024-05-09T04:59:57.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.718+0000 7f41ef7fe640 1 -- 172.21.15.114:0/2865605459 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41e800cc30 con 0x7f41f80b84e0 2024-05-09T04:59:57.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.718+0000 7f41ef7fe640 1 -- 172.21.15.114:0/2865605459 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e8013470 con 0x7f41f80b84e0 2024-05-09T04:59:57.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.718+0000 7f41fd538640 1 -- 172.21.15.114:0/2865605459 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41f81553a0 con 0x7f41f80b84e0 2024-05-09T04:59:57.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.718+0000 7f41fd538640 1 -- 172.21.15.114:0/2865605459 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41f8155960 con 0x7f41f80b84e0 2024-05-09T04:59:57.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.718+0000 7f41ef7fe640 1 -- 172.21.15.114:0/2865605459 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f41e801a070 con 0x7f41f80b84e0 2024-05-09T04:59:57.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.718+0000 7f41ef7fe640 1 -- 172.21.15.114:0/2865605459 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f41e8013aa0 con 0x7f41f80b84e0 2024-05-09T04:59:57.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.719+0000 7f41fd538640 1 -- 172.21.15.114:0/2865605459 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41f8153e50 con 0x7f41f80b84e0 2024-05-09T04:59:57.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.721+0000 7f41ef7fe640 1 -- 172.21.15.114:0/2865605459 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f41e8013610 con 0x7f41f80b84e0 2024-05-09T04:59:57.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.806+0000 7f41fd538640 1 -- 172.21.15.114:0/2865605459 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0978499e-d158-435a-8623-18c8133c31ed", "id": 3} v 0) -- 0x7f41f8051040 con 0x7f41f80b84e0 2024-05-09T04:59:57.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.810+0000 7f41ef7fe640 1 -- 172.21.15.114:0/2865605459 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0978499e-d158-435a-8623-18c8133c31ed", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f41e8013890 con 0x7f41f80b84e0 2024-05-09T04:59:57.810 INFO:teuthology.orchestra.run.smithi114.stdout:3 2024-05-09T04:59:57.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.810+0000 7f41fd538640 1 -- 172.21.15.114:0/2865605459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41f80b84e0 msgr2=0x7f41f8157140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.810+0000 7f41fd538640 1 --2- 172.21.15.114:0/2865605459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41f80b84e0 0x7f41f8157140 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f41e800ebe0 tx=0x7f41e800c520 comp rx=0 tx=0).stop 2024-05-09T04:59:57.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.810+0000 7f41fd538640 1 -- 172.21.15.114:0/2865605459 shutdown_connections 2024-05-09T04:59:57.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.810+0000 7f41fd538640 1 --2- 172.21.15.114:0/2865605459 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41f8160160 0x7f41f8160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.810+0000 7f41fd538640 1 --2- 172.21.15.114:0/2865605459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41f80b8920 0x7f41f8154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.810+0000 7f41fd538640 1 --2- 172.21.15.114:0/2865605459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41f80b84e0 0x7f41f8157140 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.810+0000 7f41fd538640 1 -- 172.21.15.114:0/2865605459 >> 172.21.15.114:0/2865605459 conn(0x7f41f806eda0 msgr2=0x7f41f80d4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:57.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.811+0000 7f41fd538640 1 -- 172.21.15.114:0/2865605459 shutdown_connections 2024-05-09T04:59:57.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.811+0000 7f41fd538640 1 -- 172.21.15.114:0/2865605459 wait complete. 2024-05-09T04:59:57.821 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 484c05ba-6ed7-4417-b5e4-b8dceda47ce0 4 2024-05-09T04:59:57.907 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T04:59:57.907+0000 7f4fbd7ff040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T04:59:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.921+0000 7eff08409640 1 Processor -- start 2024-05-09T04:59:57.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.921+0000 7eff08409640 1 -- start start 2024-05-09T04:59:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.922+0000 7eff08409640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7eff0015cf10 0x7eff0013f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.922+0000 7eff08409640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7eff000b7be0 0x7eff000c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.922+0000 7eff08409640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff000b6710 con 0x7eff0013fab0 2024-05-09T04:59:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.922+0000 7eff08409640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7eff000b64c0 con 0x7eff000b7be0 2024-05-09T04:59:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.922+0000 7eff08409640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7eff000b69f0 con 0x7eff0015cf10 2024-05-09T04:59:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.922+0000 7eff061a5640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7eff0015cf10 0x7eff0013f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.922+0000 7eff059a4640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7eff0013fab0 0x7eff000b74d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48386/0 (socket says 172.21.15.114:48386) 2024-05-09T04:59:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.922+0000 7eff059a4640 1 -- 172.21.15.114:0/1282542511 learned_addr learned my addr 172.21.15.114:0/1282542511 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-05-09T04:59:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.922+0000 7eff069a6640 1 --2- 172.21.15.114:0/1282542511 >> v2:172.21.15.60:3300/0 conn(0x7eff000b7be0 0x7eff000c71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.922+0000 7eff051a3640 1 -- 172.21.15.114:0/1282542511 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1102295888 0 0) 0x7eff000b6710 con 0x7eff0013fab0 2024-05-09T04:59:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.922+0000 7eff051a3640 1 -- 172.21.15.114:0/1282542511 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efeec003520 con 0x7eff0013fab0 2024-05-09T04:59:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.922+0000 7eff061a5640 1 -- 172.21.15.114:0/1282542511 >> v2:172.21.15.60:3300/0 conn(0x7eff000b7be0 msgr2=0x7eff000c71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T04:59:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.922+0000 7eff061a5640 1 --2- 172.21.15.114:0/1282542511 >> v2:172.21.15.60:3300/0 conn(0x7eff000b7be0 0x7eff000c71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.922+0000 7eff061a5640 1 -- 172.21.15.114:0/1282542511 >> v1:172.21.15.114:6789/0 conn(0x7eff0013fab0 legacy=0x7eff000b74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.922+0000 7eff061a5640 1 -- 172.21.15.114:0/1282542511 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff000b6cd0 con 0x7eff0015cf10 2024-05-09T04:59:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.923+0000 7eff061a5640 1 --2- 172.21.15.114:0/1282542511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff0015cf10 0x7eff0013f570 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7eff000cf3d0 tx=0x7efefc02a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14db012e98e979fc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.923+0000 7eff051a3640 1 -- 172.21.15.114:0/1282542511 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efefc007740 con 0x7eff0015cf10 2024-05-09T04:59:57.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.923+0000 7eff051a3640 1 -- 172.21.15.114:0/1282542511 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efefc0078e0 con 0x7eff0015cf10 2024-05-09T04:59:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.923+0000 7eff051a3640 1 -- 172.21.15.114:0/1282542511 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efefc007cb0 con 0x7eff0015cf10 2024-05-09T04:59:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.924+0000 7eff08409640 1 -- 172.21.15.114:0/1282542511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff0015cf10 msgr2=0x7eff0013f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.924+0000 7eff08409640 1 --2- 172.21.15.114:0/1282542511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff0015cf10 0x7eff0013f570 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7eff000cf3d0 tx=0x7efefc02a5f0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.924+0000 7eff08409640 1 -- 172.21.15.114:0/1282542511 shutdown_connections 2024-05-09T04:59:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.924+0000 7eff08409640 1 --2- 172.21.15.114:0/1282542511 >> v2:172.21.15.60:3300/0 conn(0x7eff000b7be0 0x7eff000c71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.924+0000 7eff08409640 1 --2- 172.21.15.114:0/1282542511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff0015cf10 0x7eff0013f570 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.924+0000 7eff08409640 1 -- 172.21.15.114:0/1282542511 >> 172.21.15.114:0/1282542511 conn(0x7eff0006fd00 msgr2=0x7eff00070110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.924+0000 7eff08409640 1 -- 172.21.15.114:0/1282542511 shutdown_connections 2024-05-09T04:59:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.924+0000 7eff08409640 1 -- 172.21.15.114:0/1282542511 wait complete. 2024-05-09T04:59:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.925+0000 7eff08409640 1 Processor -- start 2024-05-09T04:59:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.925+0000 7eff08409640 1 -- start start 2024-05-09T04:59:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.926+0000 7eff08409640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff000b7be0 0x7eff001f4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.926+0000 7eff08409640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff0013fab0 0x7eff00158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.926+0000 7eff08409640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff0015cf10 0x7eff0006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.926+0000 7eff08409640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7eff0015d470 con 0x7eff000b7be0 2024-05-09T04:59:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.926+0000 7eff08409640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7eff0015d2f0 con 0x7eff0013fab0 2024-05-09T04:59:57.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.926+0000 7eff08409640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7eff000c7710 con 0x7eff0015cf10 2024-05-09T04:59:57.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.926+0000 7eff059a4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff0013fab0 0x7eff00158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.926+0000 7eff061a5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff000b7be0 0x7eff001f4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.928 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:57.926+0000 7f74e271c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T04:59:57.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.926+0000 7eff059a4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff0013fab0 0x7eff00158310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52228/0 (socket says 172.21.15.114:52228) 2024-05-09T04:59:57.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.926+0000 7eff059a4640 1 -- 172.21.15.114:0/2971263068 learned_addr learned my addr 172.21.15.114:0/2971263068 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T04:59:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.926+0000 7eff069a6640 1 --2- 172.21.15.114:0/2971263068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff0015cf10 0x7eff0006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.926+0000 7eff069a6640 1 -- 172.21.15.114:0/2971263068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff0013fab0 msgr2=0x7eff00158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.926+0000 7eff069a6640 1 --2- 172.21.15.114:0/2971263068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff0013fab0 0x7eff00158310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.926+0000 7eff069a6640 1 -- 172.21.15.114:0/2971263068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff000b7be0 msgr2=0x7eff001f4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.926+0000 7eff069a6640 1 --2- 172.21.15.114:0/2971263068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff000b7be0 0x7eff001f4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.926+0000 7eff069a6640 1 -- 172.21.15.114:0/2971263068 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff00158d40 con 0x7eff0015cf10 2024-05-09T04:59:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.926+0000 7eff069a6640 1 --2- 172.21.15.114:0/2971263068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff0015cf10 0x7eff0006c370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efef800e7f0 tx=0x7efef800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.926+0000 7efef6ffd640 1 -- 172.21.15.114:0/2971263068 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef8017070 con 0x7eff0015cf10 2024-05-09T04:59:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.927+0000 7efef6ffd640 1 -- 172.21.15.114:0/2971263068 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efef800c940 con 0x7eff0015cf10 2024-05-09T04:59:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.927+0000 7eff08409640 1 -- 172.21.15.114:0/2971263068 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff00159020 con 0x7eff0015cf10 2024-05-09T04:59:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.927+0000 7efef6ffd640 1 -- 172.21.15.114:0/2971263068 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef800cd10 con 0x7eff0015cf10 2024-05-09T04:59:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.927+0000 7eff08409640 1 -- 172.21.15.114:0/2971263068 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7eff001595e0 con 0x7eff0015cf10 2024-05-09T04:59:57.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.927+0000 7efef6ffd640 1 -- 172.21.15.114:0/2971263068 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efef801a070 con 0x7eff0015cf10 2024-05-09T04:59:57.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.927+0000 7efef6ffd640 1 -- 172.21.15.114:0/2971263068 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7efef8003610 con 0x7eff0015cf10 2024-05-09T04:59:57.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.928+0000 7eff08409640 1 -- 172.21.15.114:0/2971263068 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff0015ed50 con 0x7eff0015cf10 2024-05-09T04:59:57.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:57.929+0000 7efef6ffd640 1 -- 172.21.15.114:0/2971263068 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7eff0015ed50 con 0x7eff0015cf10 2024-05-09T04:59:57.999 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T04:59:57.999+0000 7f4fbd7ff040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T04:59:58.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.002+0000 7eff08409640 1 -- 172.21.15.114:0/2971263068 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd new", "uuid": "484c05ba-6ed7-4417-b5e4-b8dceda47ce0", "id": 4} v 0) -- 0x7eff00051040 con 0x7eff0015cf10 2024-05-09T04:59:58.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.005+0000 7efef6ffd640 1 -- 172.21.15.114:0/2971263068 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "484c05ba-6ed7-4417-b5e4-b8dceda47ce0", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7efef8003bb0 con 0x7eff0015cf10 2024-05-09T04:59:58.006 INFO:teuthology.orchestra.run.smithi114.stdout:4 2024-05-09T04:59:58.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.006+0000 7eff08409640 1 -- 172.21.15.114:0/2971263068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff0015cf10 msgr2=0x7eff0006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.006+0000 7eff08409640 1 --2- 172.21.15.114:0/2971263068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff0015cf10 0x7eff0006c370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efef800e7f0 tx=0x7efef800ecb0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.006+0000 7eff08409640 1 -- 172.21.15.114:0/2971263068 shutdown_connections 2024-05-09T04:59:58.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.006+0000 7eff08409640 1 --2- 172.21.15.114:0/2971263068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff0015cf10 0x7eff0006c370 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.006+0000 7eff08409640 1 --2- 172.21.15.114:0/2971263068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff0013fab0 0x7eff00158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.006+0000 7eff08409640 1 --2- 172.21.15.114:0/2971263068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff000b7be0 0x7eff001f4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.006+0000 7eff08409640 1 -- 172.21.15.114:0/2971263068 >> 172.21.15.114:0/2971263068 conn(0x7eff0006fd00 msgr2=0x7eff00141980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:58.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.006+0000 7eff08409640 1 -- 172.21.15.114:0/2971263068 shutdown_connections 2024-05-09T04:59:58.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.006+0000 7eff08409640 1 -- 172.21.15.114:0/2971263068 wait complete. 2024-05-09T04:59:58.017 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 42c3b489-393d-4bbe-9045-06609caac26f 5 2024-05-09T04:59:58.019 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:58.019+0000 7f74e271c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T04:59:58.045 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T04:59:58.044+0000 7f4fbd7ff040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T04:59:58.065 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:58.064+0000 7f74e271c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T04:59:58.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.116+0000 7f6c7a390640 1 Processor -- start 2024-05-09T04:59:58.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.116+0000 7f6c7a390640 1 -- start start 2024-05-09T04:59:58.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.117+0000 7f6c7a390640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c740c71d0 0x7f6c740c7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.117+0000 7f6c7a390640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6c7413f7c0 0x7f6c7413fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.117+0000 7f6c7a390640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c740b7030 con 0x7f6c740b4f10 2024-05-09T04:59:58.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.117+0000 7f6c7a390640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f6c740c7b70 con 0x7f6c7413f7c0 2024-05-09T04:59:58.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.117+0000 7f6c7a390640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6c740b7c40 con 0x7f6c740c71d0 2024-05-09T04:59:58.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.117+0000 7f6c7892d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6c7413f7c0 0x7f6c7413fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.117+0000 7f6c737fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c740c71d0 0x7f6c740c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.117+0000 7f6c7892d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6c7413f7c0 0x7f6c7413fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52240/0 (socket says 172.21.15.114:52240) 2024-05-09T04:59:58.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.117+0000 7f6c7892d640 1 -- 172.21.15.114:0/3259758036 learned_addr learned my addr 172.21.15.114:0/3259758036 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T04:59:58.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.117+0000 7f6c737fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c740c71d0 0x7f6c740c7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:39962/0 (socket says 172.21.15.114:39962) 2024-05-09T04:59:58.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.117+0000 7f6c737fe640 1 -- 172.21.15.114:0/3259758036 >> v2:172.21.15.114:3300/0 conn(0x7f6c7413f7c0 msgr2=0x7f6c7413fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T04:59:58.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.117+0000 7f6c737fe640 1 --2- 172.21.15.114:0/3259758036 >> v2:172.21.15.114:3300/0 conn(0x7f6c7413f7c0 0x7f6c7413fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.117+0000 7f6c737fe640 1 -- 172.21.15.114:0/3259758036 >> v1:172.21.15.60:6789/0 conn(0x7f6c740b4f10 legacy=0x7f6c740b64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.117+0000 7f6c737fe640 1 -- 172.21.15.114:0/3259758036 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c740b7310 con 0x7f6c740c71d0 2024-05-09T04:59:58.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.117+0000 7f6c7892d640 1 --2- 172.21.15.114:0/3259758036 >> v2:172.21.15.114:3300/0 conn(0x7f6c7413f7c0 0x7f6c7413fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T04:59:58.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.118+0000 7f6c737fe640 1 --2- 172.21.15.114:0/3259758036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c740c71d0 0x7f6c740c7630 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c0097a0 tx=0x7f6c5c02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f49e06c613291449 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:58.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.118+0000 7f6c72ffd640 1 -- 172.21.15.114:0/3259758036 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c037070 con 0x7f6c740c71d0 2024-05-09T04:59:58.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.118+0000 7f6c72ffd640 1 -- 172.21.15.114:0/3259758036 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c5c02ae50 con 0x7f6c740c71d0 2024-05-09T04:59:58.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.118+0000 7f6c72ffd640 1 -- 172.21.15.114:0/3259758036 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c032660 con 0x7f6c740c71d0 2024-05-09T04:59:58.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.119+0000 7f6c7a390640 1 -- 172.21.15.114:0/3259758036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c740c71d0 msgr2=0x7f6c740c7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.119+0000 7f6c7a390640 1 --2- 172.21.15.114:0/3259758036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c740c71d0 0x7f6c740c7630 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c0097a0 tx=0x7f6c5c02a100 comp rx=0 tx=0).stop 2024-05-09T04:59:58.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.119+0000 7f6c7a390640 1 -- 172.21.15.114:0/3259758036 shutdown_connections 2024-05-09T04:59:58.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.119+0000 7f6c7a390640 1 --2- 172.21.15.114:0/3259758036 >> v2:172.21.15.114:3300/0 conn(0x7f6c7413f7c0 0x7f6c7413fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.119+0000 7f6c7a390640 1 --2- 172.21.15.114:0/3259758036 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c740c71d0 0x7f6c740c7630 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.119+0000 7f6c7a390640 1 -- 172.21.15.114:0/3259758036 >> 172.21.15.114:0/3259758036 conn(0x7f6c7406ff10 msgr2=0x7f6c74070320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:58.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.119+0000 7f6c7a390640 1 -- 172.21.15.114:0/3259758036 shutdown_connections 2024-05-09T04:59:58.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.119+0000 7f6c7a390640 1 -- 172.21.15.114:0/3259758036 wait complete. 2024-05-09T04:59:58.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.120+0000 7f6c7a390640 1 Processor -- start 2024-05-09T04:59:58.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.120+0000 7f6c7a390640 1 -- start start 2024-05-09T04:59:58.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.121+0000 7f6c7a390640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c740b4f10 0x7f6c7415c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.121+0000 7f6c7a390640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c740c71d0 0x7f6c7414f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.121+0000 7f6c7a390640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c7413f7c0 0x7f6c7406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.121+0000 7f6c7a390640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6c740cf910 con 0x7f6c7413f7c0 2024-05-09T04:59:58.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.121+0000 7f6c7a390640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6c740b78d0 con 0x7f6c740b4f10 2024-05-09T04:59:58.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.121+0000 7f6c7a390640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6c740b6bd0 con 0x7f6c740c71d0 2024-05-09T04:59:58.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.121+0000 7f6c73fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c740b4f10 0x7f6c7415c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.121+0000 7f6c737fe640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c740c71d0 0x7f6c7414f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.121+0000 7f6c73fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c740b4f10 0x7f6c7415c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52254/0 (socket says 172.21.15.114:52254) 2024-05-09T04:59:58.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.121+0000 7f6c73fff640 1 -- 172.21.15.114:0/2248860844 learned_addr learned my addr 172.21.15.114:0/2248860844 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T04:59:58.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.121+0000 7f6c7892d640 1 --2- 172.21.15.114:0/2248860844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c7413f7c0 0x7f6c7406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.121+0000 7f6c73fff640 1 -- 172.21.15.114:0/2248860844 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c740c71d0 msgr2=0x7f6c7414f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.121+0000 7f6c73fff640 1 --2- 172.21.15.114:0/2248860844 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c740c71d0 0x7f6c7414f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.121+0000 7f6c73fff640 1 -- 172.21.15.114:0/2248860844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c7413f7c0 msgr2=0x7f6c7406c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.121+0000 7f6c73fff640 1 --2- 172.21.15.114:0/2248860844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c7413f7c0 0x7f6c7406c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.121+0000 7f6c73fff640 1 -- 172.21.15.114:0/2248860844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c7413ef20 con 0x7f6c740b4f10 2024-05-09T04:59:58.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.121+0000 7f6c73fff640 1 --2- 172.21.15.114:0/2248860844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c740b4f10 0x7f6c7415c680 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6c6400c500 tx=0x7f6c6400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:58.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.121+0000 7f6c70ff9640 1 -- 172.21.15.114:0/2248860844 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c640052d0 con 0x7f6c740b4f10 2024-05-09T04:59:58.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.122+0000 7f6c70ff9640 1 -- 172.21.15.114:0/2248860844 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c64005470 con 0x7f6c740b4f10 2024-05-09T04:59:58.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.122+0000 7f6c70ff9640 1 -- 172.21.15.114:0/2248860844 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c64012870 con 0x7f6c740b4f10 2024-05-09T04:59:58.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.122+0000 7f6c7a390640 1 -- 172.21.15.114:0/2248860844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c7415cd80 con 0x7f6c740b4f10 2024-05-09T04:59:58.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.122+0000 7f6c7a390640 1 -- 172.21.15.114:0/2248860844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c7415d340 con 0x7f6c740b4f10 2024-05-09T04:59:58.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.122+0000 7f6c70ff9640 1 -- 172.21.15.114:0/2248860844 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6c64012a10 con 0x7f6c740b4f10 2024-05-09T04:59:58.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.122+0000 7f6c70ff9640 1 -- 172.21.15.114:0/2248860844 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f6c6401bae0 con 0x7f6c740b4f10 2024-05-09T04:59:58.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.122+0000 7f6c7a390640 1 -- 172.21.15.114:0/2248860844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c74051040 con 0x7f6c740b4f10 2024-05-09T04:59:58.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.124+0000 7f6c70ff9640 1 -- 172.21.15.114:0/2248860844 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f6c6401b520 con 0x7f6c740b4f10 2024-05-09T04:59:58.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.196+0000 7f6c7a390640 1 -- 172.21.15.114:0/2248860844 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd new", "uuid": "42c3b489-393d-4bbe-9045-06609caac26f", "id": 5} v 0) -- 0x7f6c7413f3a0 con 0x7f6c740b4f10 2024-05-09T04:59:58.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.200+0000 7f6c70ff9640 1 -- 172.21.15.114:0/2248860844 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "42c3b489-393d-4bbe-9045-06609caac26f", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f6c6401b700 con 0x7f6c740b4f10 2024-05-09T04:59:58.200 INFO:teuthology.orchestra.run.smithi114.stdout:5 2024-05-09T04:59:58.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.201+0000 7f6c7a390640 1 -- 172.21.15.114:0/2248860844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c740b4f10 msgr2=0x7f6c7415c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.201+0000 7f6c7a390640 1 --2- 172.21.15.114:0/2248860844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c740b4f10 0x7f6c7415c680 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6c6400c500 tx=0x7f6c6400c9c0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.201+0000 7f6c7a390640 1 -- 172.21.15.114:0/2248860844 shutdown_connections 2024-05-09T04:59:58.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.201+0000 7f6c7a390640 1 --2- 172.21.15.114:0/2248860844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c7413f7c0 0x7f6c7406c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.201+0000 7f6c7a390640 1 --2- 172.21.15.114:0/2248860844 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6c740c71d0 0x7f6c7414f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.201+0000 7f6c7a390640 1 --2- 172.21.15.114:0/2248860844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c740b4f10 0x7f6c7415c680 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.201+0000 7f6c7a390640 1 -- 172.21.15.114:0/2248860844 >> 172.21.15.114:0/2248860844 conn(0x7f6c7406ff10 msgr2=0x7f6c740b5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:58.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.201+0000 7f6c7a390640 1 -- 172.21.15.114:0/2248860844 shutdown_connections 2024-05-09T04:59:58.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.201+0000 7f6c7a390640 1 -- 172.21.15.114:0/2248860844 wait complete. 2024-05-09T04:59:58.211 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 335cac6a-28ba-449c-9f6d-721790845461 6 2024-05-09T04:59:58.219 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T04:59:58.219+0000 7f4fbd7ff040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T04:59:58.262 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:58.261+0000 7f74e271c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T04:59:58.270 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T04:59:58.269+0000 7f4fbd7ff040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T04:59:58.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.310+0000 7fd604059640 1 Processor -- start 2024-05-09T04:59:58.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.310+0000 7fd604059640 1 -- start start 2024-05-09T04:59:58.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.311+0000 7fd604059640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5fc160160 0x7fd5fc160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.311+0000 7fd604059640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd5fc0b84e0 0x7fd5fc0ba230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.311+0000 7fd604059640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd5fc0b8920 0x7fd5fc0ba770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.311+0000 7fd604059640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fd5fc0b9790 con 0x7fd5fc0b84e0 2024-05-09T04:59:58.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.311+0000 7fd604059640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fd5fc0b8ef0 con 0x7fd5fc0b8920 2024-05-09T04:59:58.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.311+0000 7fd604059640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd5fc0b91d0 con 0x7fd5fc160160 2024-05-09T04:59:58.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.311+0000 7fd601df5640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5fc160160 0x7fd5fc160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.311+0000 7fd6025f6640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd5fc0b8920 0x7fd5fc0ba770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.311+0000 7fd601df5640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5fc160160 0x7fd5fc160540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:39968/0 (socket says 172.21.15.114:39968) 2024-05-09T04:59:58.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.311+0000 7fd601df5640 1 -- 172.21.15.114:0/3503605913 learned_addr learned my addr 172.21.15.114:0/3503605913 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T04:59:58.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.311+0000 7fd6015f4640 1 --2- 172.21.15.114:0/3503605913 >> v2:172.21.15.60:3300/0 conn(0x7fd5fc0b84e0 0x7fd5fc0ba230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.311+0000 7fd601df5640 1 -- 172.21.15.114:0/3503605913 >> v2:172.21.15.114:3300/0 conn(0x7fd5fc0b8920 msgr2=0x7fd5fc0ba770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T04:59:58.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.311+0000 7fd601df5640 1 --2- 172.21.15.114:0/3503605913 >> v2:172.21.15.114:3300/0 conn(0x7fd5fc0b8920 0x7fd5fc0ba770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.311+0000 7fd601df5640 1 -- 172.21.15.114:0/3503605913 >> v2:172.21.15.60:3300/0 conn(0x7fd5fc0b84e0 msgr2=0x7fd5fc0ba230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T04:59:58.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.311+0000 7fd601df5640 1 --2- 172.21.15.114:0/3503605913 >> v2:172.21.15.60:3300/0 conn(0x7fd5fc0b84e0 0x7fd5fc0ba230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.311+0000 7fd601df5640 1 -- 172.21.15.114:0/3503605913 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5fc0b94b0 con 0x7fd5fc160160 2024-05-09T04:59:58.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.311+0000 7fd6025f6640 1 --2- 172.21.15.114:0/3503605913 >> v2:172.21.15.114:3300/0 conn(0x7fd5fc0b8920 0x7fd5fc0ba770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T04:59:58.313 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:58.312+0000 7f74e271c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T04:59:58.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.312+0000 7fd601df5640 1 --2- 172.21.15.114:0/3503605913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5fc160160 0x7fd5fc160540 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc0d20b0 tx=0x7fd5f802a140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5eb1b0864cef858 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:58.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.312+0000 7fd600df3640 1 -- 172.21.15.114:0/3503605913 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8037070 con 0x7fd5fc160160 2024-05-09T04:59:58.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.312+0000 7fd600df3640 1 -- 172.21.15.114:0/3503605913 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5f8032440 con 0x7fd5fc160160 2024-05-09T04:59:58.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.312+0000 7fd600df3640 1 -- 172.21.15.114:0/3503605913 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8032810 con 0x7fd5fc160160 2024-05-09T04:59:58.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.313+0000 7fd604059640 1 -- 172.21.15.114:0/3503605913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5fc160160 msgr2=0x7fd5fc160540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.313+0000 7fd604059640 1 --2- 172.21.15.114:0/3503605913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5fc160160 0x7fd5fc160540 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc0d20b0 tx=0x7fd5f802a140 comp rx=0 tx=0).stop 2024-05-09T04:59:58.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.313+0000 7fd604059640 1 -- 172.21.15.114:0/3503605913 shutdown_connections 2024-05-09T04:59:58.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.313+0000 7fd604059640 1 --2- 172.21.15.114:0/3503605913 >> v2:172.21.15.114:3300/0 conn(0x7fd5fc0b8920 0x7fd5fc0ba770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.313+0000 7fd604059640 1 --2- 172.21.15.114:0/3503605913 >> v2:172.21.15.60:3300/0 conn(0x7fd5fc0b84e0 0x7fd5fc0ba230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.313+0000 7fd604059640 1 --2- 172.21.15.114:0/3503605913 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5fc160160 0x7fd5fc160540 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.313+0000 7fd604059640 1 -- 172.21.15.114:0/3503605913 >> 172.21.15.114:0/3503605913 conn(0x7fd5fc06eda0 msgr2=0x7fd5fc073820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.313+0000 7fd604059640 1 -- 172.21.15.114:0/3503605913 shutdown_connections 2024-05-09T04:59:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.313+0000 7fd604059640 1 -- 172.21.15.114:0/3503605913 wait complete. 2024-05-09T04:59:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.314+0000 7fd604059640 1 Processor -- start 2024-05-09T04:59:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.314+0000 7fd604059640 1 -- start start 2024-05-09T04:59:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.315+0000 7fd604059640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5fc0b84e0 0x7fd5fc157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.315+0000 7fd604059640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5fc0b8920 0x7fd5fc0afe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.315+0000 7fd604059640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5fc160160 0x7fd5fc0b0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.315+0000 7fd604059640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc15fd90 con 0x7fd5fc0b84e0 2024-05-09T04:59:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.315+0000 7fd604059640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc15fc10 con 0x7fd5fc160160 2024-05-09T04:59:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.315+0000 7fd604059640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd5fc162fd0 con 0x7fd5fc0b8920 2024-05-09T04:59:58.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.315+0000 7fd6015f4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5fc0b8920 0x7fd5fc0afe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.315+0000 7fd6025f6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5fc160160 0x7fd5fc0b0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.315+0000 7fd6015f4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5fc0b8920 0x7fd5fc0afe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:39984/0 (socket says 172.21.15.114:39984) 2024-05-09T04:59:58.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.315+0000 7fd6015f4640 1 -- 172.21.15.114:0/3633426331 learned_addr learned my addr 172.21.15.114:0/3633426331 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T04:59:58.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.315+0000 7fd601df5640 1 --2- 172.21.15.114:0/3633426331 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5fc0b84e0 0x7fd5fc157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.315+0000 7fd6015f4640 1 -- 172.21.15.114:0/3633426331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5fc160160 msgr2=0x7fd5fc0b0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.315+0000 7fd6015f4640 1 --2- 172.21.15.114:0/3633426331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5fc160160 0x7fd5fc0b0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.315+0000 7fd6015f4640 1 -- 172.21.15.114:0/3633426331 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5fc0b84e0 msgr2=0x7fd5fc157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.315+0000 7fd6015f4640 1 --2- 172.21.15.114:0/3633426331 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5fc0b84e0 0x7fd5fc157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.315+0000 7fd6015f4640 1 -- 172.21.15.114:0/3633426331 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5fc15e6e0 con 0x7fd5fc0b8920 2024-05-09T04:59:58.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.315+0000 7fd601df5640 1 --2- 172.21.15.114:0/3633426331 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5fc0b84e0 0x7fd5fc157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T04:59:58.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.315+0000 7fd6015f4640 1 --2- 172.21.15.114:0/3633426331 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5fc0b8920 0x7fd5fc0afe20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00e8a0 tx=0x7fd5ec00eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:58.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.316+0000 7fd5f27fc640 1 -- 172.21.15.114:0/3633426331 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec017070 con 0x7fd5fc0b8920 2024-05-09T04:59:58.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.316+0000 7fd5f27fc640 1 -- 172.21.15.114:0/3633426331 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5ec00c730 con 0x7fd5fc0b8920 2024-05-09T04:59:58.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.316+0000 7fd5f27fc640 1 -- 172.21.15.114:0/3633426331 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec00cb20 con 0x7fd5fc0b8920 2024-05-09T04:59:58.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.316+0000 7fd604059640 1 -- 172.21.15.114:0/3633426331 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5fc15e9c0 con 0x7fd5fc0b8920 2024-05-09T04:59:58.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.316+0000 7fd604059640 1 -- 172.21.15.114:0/3633426331 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5fc15b290 con 0x7fd5fc0b8920 2024-05-09T04:59:58.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.316+0000 7fd5f27fc640 1 -- 172.21.15.114:0/3633426331 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd5ec01a070 con 0x7fd5fc0b8920 2024-05-09T04:59:58.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.316+0000 7fd5f27fc640 1 -- 172.21.15.114:0/3633426331 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fd5ec013dd0 con 0x7fd5fc0b8920 2024-05-09T04:59:58.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.317+0000 7fd604059640 1 -- 172.21.15.114:0/3633426331 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5fc15eee0 con 0x7fd5fc0b8920 2024-05-09T04:59:58.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.320+0000 7fd5f27fc640 1 -- 172.21.15.114:0/3633426331 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fd5fc15eee0 con 0x7fd5fc0b8920 2024-05-09T04:59:58.325 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T04:59:58.325+0000 7f4fbd7ff040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T04:59:58.369 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:58.369+0000 7f74e271c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T04:59:58.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.393+0000 7fd604059640 1 -- 172.21.15.114:0/3633426331 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd new", "uuid": "335cac6a-28ba-449c-9f6d-721790845461", "id": 6} v 0) -- 0x7fd5fc0d4e50 con 0x7fd5fc0b8920 2024-05-09T04:59:58.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.396+0000 7fd5f27fc640 1 -- 172.21.15.114:0/3633426331 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "335cac6a-28ba-449c-9f6d-721790845461", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fd5ec01c450 con 0x7fd5fc0b8920 2024-05-09T04:59:58.397 INFO:teuthology.orchestra.run.smithi114.stdout:6 2024-05-09T04:59:58.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.397+0000 7fd604059640 1 -- 172.21.15.114:0/3633426331 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5fc0b8920 msgr2=0x7fd5fc0afe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.397+0000 7fd604059640 1 --2- 172.21.15.114:0/3633426331 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5fc0b8920 0x7fd5fc0afe20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00e8a0 tx=0x7fd5ec00eca0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.397+0000 7fd604059640 1 -- 172.21.15.114:0/3633426331 shutdown_connections 2024-05-09T04:59:58.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.397+0000 7fd604059640 1 --2- 172.21.15.114:0/3633426331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5fc160160 0x7fd5fc0b0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.397+0000 7fd604059640 1 --2- 172.21.15.114:0/3633426331 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd5fc0b8920 0x7fd5fc0afe20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.397+0000 7fd604059640 1 --2- 172.21.15.114:0/3633426331 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd5fc0b84e0 0x7fd5fc157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.397+0000 7fd604059640 1 -- 172.21.15.114:0/3633426331 >> 172.21.15.114:0/3633426331 conn(0x7fd5fc06eda0 msgr2=0x7fd5fc0737b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:58.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.397+0000 7fd604059640 1 -- 172.21.15.114:0/3633426331 shutdown_connections 2024-05-09T04:59:58.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.397+0000 7fd604059640 1 -- 172.21.15.114:0/3633426331 wait complete. 2024-05-09T04:59:58.408 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd new 8ed03228-ebba-492d-9d72-936b27450f14 7 2024-05-09T04:59:58.444 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T04:59:58.444+0000 7f4fbd7ff040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T04:59:58.489 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:58.489+0000 7f74e271c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T04:59:58.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.508+0000 7f6daa79d640 1 Processor -- start 2024-05-09T04:59:58.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.508+0000 7f6daa79d640 1 -- start start 2024-05-09T04:59:58.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.509+0000 7f6daa79d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6da40b64c0 0x7f6da4072130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.509+0000 7f6daa79d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6da4072670 0x7f6da40b4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.509+0000 7f6daa79d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6da40b6950 con 0x7f6da4160430 2024-05-09T04:59:58.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.509+0000 7f6daa79d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f6da4072ac0 con 0x7f6da40b64c0 2024-05-09T04:59:58.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.509+0000 7f6daa79d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6da40b6c30 con 0x7f6da4072670 2024-05-09T04:59:58.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.509+0000 7f6da3fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f6da4160430 0x7f6da4163780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48390/0 (socket says 172.21.15.114:48390) 2024-05-09T04:59:58.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.509+0000 7f6da3fff640 1 -- 172.21.15.114:0/1945915802 learned_addr learned my addr 172.21.15.114:0/1945915802 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-05-09T04:59:58.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.509+0000 7f6da8d3a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6da4072670 0x7f6da40b4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.509+0000 7f6da37fe640 1 --2- 172.21.15.114:0/1945915802 >> v2:172.21.15.60:3300/0 conn(0x7f6da40b64c0 0x7f6da4072130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.509+0000 7f6da8d3a640 1 -- 172.21.15.114:0/1945915802 >> v2:172.21.15.60:3300/0 conn(0x7f6da40b64c0 msgr2=0x7f6da4072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T04:59:58.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.509+0000 7f6da8d3a640 1 --2- 172.21.15.114:0/1945915802 >> v2:172.21.15.60:3300/0 conn(0x7f6da40b64c0 0x7f6da4072130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.509+0000 7f6da2ffd640 1 -- 172.21.15.114:0/1945915802 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3719043231 0 0) 0x7f6da40b6950 con 0x7f6da4160430 2024-05-09T04:59:58.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.509+0000 7f6da8d3a640 1 -- 172.21.15.114:0/1945915802 >> v1:172.21.15.114:6789/0 conn(0x7f6da4160430 legacy=0x7f6da4163780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.509+0000 7f6da8d3a640 1 -- 172.21.15.114:0/1945915802 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da40b6f10 con 0x7f6da4072670 2024-05-09T04:59:58.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.510+0000 7f6da8d3a640 1 --2- 172.21.15.114:0/1945915802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6da4072670 0x7f6da40b4520 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009d90 tx=0x7f6d9c02a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36121f99676969a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:58.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.510+0000 7f6da2ffd640 1 -- 172.21.15.114:0/1945915802 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c031600 con 0x7f6da4072670 2024-05-09T04:59:58.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.510+0000 7f6da2ffd640 1 -- 172.21.15.114:0/1945915802 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d9c0317a0 con 0x7f6da4072670 2024-05-09T04:59:58.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.510+0000 7f6da2ffd640 1 -- 172.21.15.114:0/1945915802 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c031b70 con 0x7f6da4072670 2024-05-09T04:59:58.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.511+0000 7f6daa79d640 1 -- 172.21.15.114:0/1945915802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6da4072670 msgr2=0x7f6da40b4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.511+0000 7f6daa79d640 1 --2- 172.21.15.114:0/1945915802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6da4072670 0x7f6da40b4520 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009d90 tx=0x7f6d9c02a5d0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.511+0000 7f6daa79d640 1 -- 172.21.15.114:0/1945915802 shutdown_connections 2024-05-09T04:59:58.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.511+0000 7f6daa79d640 1 --2- 172.21.15.114:0/1945915802 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6da4072670 0x7f6da40b4520 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.511+0000 7f6daa79d640 1 --2- 172.21.15.114:0/1945915802 >> v2:172.21.15.60:3300/0 conn(0x7f6da40b64c0 0x7f6da4072130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.511+0000 7f6daa79d640 1 -- 172.21.15.114:0/1945915802 >> 172.21.15.114:0/1945915802 conn(0x7f6da406fd00 msgr2=0x7f6da4070110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:58.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.511+0000 7f6daa79d640 1 -- 172.21.15.114:0/1945915802 shutdown_connections 2024-05-09T04:59:58.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.511+0000 7f6daa79d640 1 -- 172.21.15.114:0/1945915802 wait complete. 2024-05-09T04:59:58.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.512+0000 7f6daa79d640 1 Processor -- start 2024-05-09T04:59:58.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.512+0000 7f6daa79d640 1 -- start start 2024-05-09T04:59:58.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.513+0000 7f6daa79d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6da4072670 0x7f6da415ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.513+0000 7f6daa79d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6da4160430 0x7f6da41f41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.513+0000 7f6daa79d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6da41468e0 0x7f6da4146d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.513+0000 7f6daa79d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6da40b4a60 con 0x7f6da4072670 2024-05-09T04:59:58.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.513+0000 7f6daa79d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6da415faa0 con 0x7f6da4160430 2024-05-09T04:59:58.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.513+0000 7f6daa79d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6da40b4be0 con 0x7f6da41468e0 2024-05-09T04:59:58.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.513+0000 7f6da8d3a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6da41468e0 0x7f6da4146d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.513+0000 7f6da8d3a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6da41468e0 0x7f6da4146d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.114:40004/0 (socket says 172.21.15.114:40004) 2024-05-09T04:59:58.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.513+0000 7f6da37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6da4160430 0x7f6da41f41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.513+0000 7f6da8d3a640 1 -- 172.21.15.114:0/575097230 learned_addr learned my addr 172.21.15.114:0/575097230 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-05-09T04:59:58.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.513+0000 7f6da3fff640 1 --2- 172.21.15.114:0/575097230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6da4072670 0x7f6da415ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.513+0000 7f6da8d3a640 1 -- 172.21.15.114:0/575097230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6da4160430 msgr2=0x7f6da41f41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.513+0000 7f6da8d3a640 1 --2- 172.21.15.114:0/575097230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6da4160430 0x7f6da41f41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.513+0000 7f6da8d3a640 1 -- 172.21.15.114:0/575097230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6da4072670 msgr2=0x7f6da415ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.513+0000 7f6da8d3a640 1 --2- 172.21.15.114:0/575097230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6da4072670 0x7f6da415ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.513+0000 7f6da8d3a640 1 -- 172.21.15.114:0/575097230 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da40ccba0 con 0x7f6da41468e0 2024-05-09T04:59:58.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.513+0000 7f6da3fff640 1 --2- 172.21.15.114:0/575097230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6da4072670 0x7f6da415ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T04:59:58.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.513+0000 7f6da37fe640 1 --2- 172.21.15.114:0/575097230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6da4160430 0x7f6da41f41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T04:59:58.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.514+0000 7f6da8d3a640 1 --2- 172.21.15.114:0/575097230 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6da41468e0 0x7f6da4146d90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009d90 tx=0x7f6d9c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:58.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.514+0000 7f6da0ff9640 1 -- 172.21.15.114:0/575097230 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c040070 con 0x7f6da41468e0 2024-05-09T04:59:58.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.514+0000 7f6da0ff9640 1 -- 172.21.15.114:0/575097230 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d9c030be0 con 0x7f6da41468e0 2024-05-09T04:59:58.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.514+0000 7f6da0ff9640 1 -- 172.21.15.114:0/575097230 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c009450 con 0x7f6da41468e0 2024-05-09T04:59:58.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.514+0000 7f6daa79d640 1 -- 172.21.15.114:0/575097230 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6da40cc5e0 con 0x7f6da41468e0 2024-05-09T04:59:58.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.514+0000 7f6daa79d640 1 -- 172.21.15.114:0/575097230 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6da415a2f0 con 0x7f6da41468e0 2024-05-09T04:59:58.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.514+0000 7f6da0ff9640 1 -- 172.21.15.114:0/575097230 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6d9c039040 con 0x7f6da41468e0 2024-05-09T04:59:58.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.514+0000 7f6da0ff9640 1 -- 172.21.15.114:0/575097230 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f6d9c004790 con 0x7f6da41468e0 2024-05-09T04:59:58.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.515+0000 7f6daa79d640 1 -- 172.21.15.114:0/575097230 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6da4051040 con 0x7f6da41468e0 2024-05-09T04:59:58.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.517+0000 7f6da0ff9640 1 -- 172.21.15.114:0/575097230 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f6d9c031700 con 0x7f6da41468e0 2024-05-09T04:59:58.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.589+0000 7f6daa79d640 1 -- 172.21.15.114:0/575097230 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8ed03228-ebba-492d-9d72-936b27450f14", "id": 7} v 0) -- 0x7f6da40b7fe0 con 0x7f6da41468e0 2024-05-09T04:59:58.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.593+0000 7f6da0ff9640 1 -- 172.21.15.114:0/575097230 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8ed03228-ebba-492d-9d72-936b27450f14", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f6d9c004030 con 0x7f6da41468e0 2024-05-09T04:59:58.594 INFO:teuthology.orchestra.run.smithi114.stdout:7 2024-05-09T04:59:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.594+0000 7f6daa79d640 1 -- 172.21.15.114:0/575097230 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6da41468e0 msgr2=0x7f6da4146d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.594+0000 7f6daa79d640 1 --2- 172.21.15.114:0/575097230 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6da41468e0 0x7f6da4146d90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009d90 tx=0x7f6d9c004a40 comp rx=0 tx=0).stop 2024-05-09T04:59:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.594+0000 7f6daa79d640 1 -- 172.21.15.114:0/575097230 shutdown_connections 2024-05-09T04:59:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.594+0000 7f6daa79d640 1 --2- 172.21.15.114:0/575097230 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6da41468e0 0x7f6da4146d90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.594+0000 7f6daa79d640 1 --2- 172.21.15.114:0/575097230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6da4160430 0x7f6da41f41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.594+0000 7f6daa79d640 1 --2- 172.21.15.114:0/575097230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6da4072670 0x7f6da415ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.594+0000 7f6daa79d640 1 -- 172.21.15.114:0/575097230 >> 172.21.15.114:0/575097230 conn(0x7f6da406fd00 msgr2=0x7f6da40700e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.594+0000 7f6daa79d640 1 -- 172.21.15.114:0/575097230 shutdown_connections 2024-05-09T04:59:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T04:59:58.594+0000 7f6daa79d640 1 -- 172.21.15.114:0/575097230 wait complete. 2024-05-09T04:59:58.605 INFO:tasks.ceph.osd.0:Restarting daemon 2024-05-09T04:59:58.605 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-05-09T04:59:58.608 INFO:tasks.ceph.osd.0:Started 2024-05-09T04:59:58.608 INFO:tasks.ceph.osd.1:Restarting daemon 2024-05-09T04:59:58.608 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-05-09T04:59:58.610 INFO:tasks.ceph.osd.1:Started 2024-05-09T04:59:58.610 INFO:tasks.ceph.osd.2:Restarting daemon 2024-05-09T04:59:58.611 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-05-09T04:59:58.613 INFO:tasks.ceph.osd.2:Started 2024-05-09T04:59:58.613 INFO:tasks.ceph.osd.3:Restarting daemon 2024-05-09T04:59:58.613 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-05-09T04:59:58.615 INFO:tasks.ceph.osd.3:Started 2024-05-09T04:59:58.616 INFO:tasks.ceph.osd.4:Restarting daemon 2024-05-09T04:59:58.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-05-09T04:59:58.618 INFO:tasks.ceph.osd.4:Started 2024-05-09T04:59:58.618 INFO:tasks.ceph.osd.5:Restarting daemon 2024-05-09T04:59:58.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-05-09T04:59:58.620 INFO:tasks.ceph.osd.5:Started 2024-05-09T04:59:58.620 INFO:tasks.ceph.osd.6:Restarting daemon 2024-05-09T04:59:58.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-05-09T04:59:58.622 INFO:tasks.ceph.osd.6:Started 2024-05-09T04:59:58.622 INFO:tasks.ceph.osd.7:Restarting daemon 2024-05-09T04:59:58.623 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-05-09T04:59:58.625 INFO:tasks.ceph.osd.7:Started 2024-05-09T04:59:58.625 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T04:59:58.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.778+0000 7f8d27f32640 1 Processor -- start 2024-05-09T04:59:58.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.778+0000 7f8d27f32640 1 -- start start 2024-05-09T04:59:58.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.779+0000 7f8d27f32640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8d200aedc0 0x7f8d200702a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.779+0000 7f8d27f32640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d200b4950 0x7f8d200b4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.779+0000 7f8d27f32640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d2015fdb0 con 0x7f8d2015f890 2024-05-09T04:59:58.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.779+0000 7f8d27f32640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f8d200707e0 con 0x7f8d200aedc0 2024-05-09T04:59:58.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.779+0000 7f8d27f32640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8d200b52e0 con 0x7f8d200b4950 2024-05-09T04:59:58.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.779+0000 7f8d25cce640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8d200aedc0 0x7f8d200702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.779+0000 7f8d25cce640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8d200aedc0 0x7f8d200702a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33668/0 (socket says 172.21.15.60:33668) 2024-05-09T04:59:58.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.779+0000 7f8d25cce640 1 -- 172.21.15.60:0/3347891907 learned_addr learned my addr 172.21.15.60:0/3347891907 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T04:59:58.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.779+0000 7f8d264cf640 1 --2- 172.21.15.60:0/3347891907 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d200b4950 0x7f8d200b4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.780+0000 7f8d25cce640 1 -- 172.21.15.60:0/3347891907 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d200b4950 msgr2=0x7f8d200b4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T04:59:58.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.780+0000 7f8d25cce640 1 --2- 172.21.15.60:0/3347891907 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d200b4950 0x7f8d200b4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.780+0000 7f8d25cce640 1 -- 172.21.15.60:0/3347891907 >> v1:172.21.15.114:6789/0 conn(0x7f8d2015f890 legacy=0x7f8d200adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.780+0000 7f8d25cce640 1 -- 172.21.15.60:0/3347891907 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d20160090 con 0x7f8d200aedc0 2024-05-09T04:59:58.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.780+0000 7f8d25cce640 1 --2- 172.21.15.60:0/3347891907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d200aedc0 0x7f8d200702a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8d14009870 tx=0x7f8d1402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40addb34296e41f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:58.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.780+0000 7f8d24ccc640 1 -- 172.21.15.60:0/3347891907 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d14037070 con 0x7f8d200aedc0 2024-05-09T04:59:58.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.780+0000 7f8d24ccc640 1 -- 172.21.15.60:0/3347891907 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d14032440 con 0x7f8d200aedc0 2024-05-09T04:59:58.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.780+0000 7f8d24ccc640 1 -- 172.21.15.60:0/3347891907 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d14032810 con 0x7f8d200aedc0 2024-05-09T04:59:58.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.781+0000 7f8d27f32640 1 -- 172.21.15.60:0/3347891907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d200aedc0 msgr2=0x7f8d200702a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.781+0000 7f8d27f32640 1 --2- 172.21.15.60:0/3347891907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d200aedc0 0x7f8d200702a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8d14009870 tx=0x7f8d1402a270 comp rx=0 tx=0).stop 2024-05-09T04:59:58.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.781+0000 7f8d27f32640 1 -- 172.21.15.60:0/3347891907 shutdown_connections 2024-05-09T04:59:58.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.781+0000 7f8d27f32640 1 --2- 172.21.15.60:0/3347891907 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d200b4950 0x7f8d200b4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.781+0000 7f8d27f32640 1 --2- 172.21.15.60:0/3347891907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d200aedc0 0x7f8d200702a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.781+0000 7f8d27f32640 1 -- 172.21.15.60:0/3347891907 >> 172.21.15.60:0/3347891907 conn(0x7f8d20074780 msgr2=0x7f8d20073240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:58.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.781+0000 7f8d27f32640 1 -- 172.21.15.60:0/3347891907 shutdown_connections 2024-05-09T04:59:58.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.781+0000 7f8d27f32640 1 -- 172.21.15.60:0/3347891907 wait complete. 2024-05-09T04:59:58.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.782+0000 7f8d27f32640 1 Processor -- start 2024-05-09T04:59:58.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.782+0000 7f8d27f32640 1 -- start start 2024-05-09T04:59:58.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.783+0000 7f8d27f32640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d200aedc0 0x7f8d200ad0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.783+0000 7f8d27f32640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d200b4950 0x7f8d200cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.783+0000 7f8d27f32640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d2015f890 0x7f8d2015c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:58.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.783+0000 7f8d27f32640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8d200af1a0 con 0x7f8d200aedc0 2024-05-09T04:59:58.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.783+0000 7f8d27f32640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8d20160650 con 0x7f8d200b4950 2024-05-09T04:59:58.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.783+0000 7f8d27f32640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8d200b09c0 con 0x7f8d2015f890 2024-05-09T04:59:58.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.783+0000 7f8d25cce640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d200aedc0 0x7f8d200ad0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.783+0000 7f8d25cce640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d200aedc0 0x7f8d200ad0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33680/0 (socket says 172.21.15.60:33680) 2024-05-09T04:59:58.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.783+0000 7f8d25cce640 1 -- 172.21.15.60:0/3712568414 learned_addr learned my addr 172.21.15.60:0/3712568414 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T04:59:58.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.783+0000 7f8d254cd640 1 --2- 172.21.15.60:0/3712568414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d200b4950 0x7f8d200cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.783+0000 7f8d264cf640 1 --2- 172.21.15.60:0/3712568414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d2015f890 0x7f8d2015c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:58.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.784+0000 7f8d25cce640 1 -- 172.21.15.60:0/3712568414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d2015f890 msgr2=0x7f8d2015c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.784+0000 7f8d25cce640 1 --2- 172.21.15.60:0/3712568414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d2015f890 0x7f8d2015c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.784+0000 7f8d25cce640 1 -- 172.21.15.60:0/3712568414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d200b4950 msgr2=0x7f8d200cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.784+0000 7f8d25cce640 1 --2- 172.21.15.60:0/3712568414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d200b4950 0x7f8d200cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.784+0000 7f8d25cce640 1 -- 172.21.15.60:0/3712568414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d2015b060 con 0x7f8d200aedc0 2024-05-09T04:59:58.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.784+0000 7f8d25cce640 1 --2- 172.21.15.60:0/3712568414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d200aedc0 0x7f8d200ad0f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8d14000c00 tx=0x7f8d140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:58.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.784+0000 7f8d127fc640 1 -- 172.21.15.60:0/3712568414 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d14037040 con 0x7f8d200aedc0 2024-05-09T04:59:58.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.784+0000 7f8d127fc640 1 -- 172.21.15.60:0/3712568414 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d14042070 con 0x7f8d200aedc0 2024-05-09T04:59:58.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.784+0000 7f8d27f32640 1 -- 172.21.15.60:0/3712568414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d2015b340 con 0x7f8d200aedc0 2024-05-09T04:59:58.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.784+0000 7f8d127fc640 1 -- 172.21.15.60:0/3712568414 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d14030880 con 0x7f8d200aedc0 2024-05-09T04:59:58.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.784+0000 7f8d27f32640 1 -- 172.21.15.60:0/3712568414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d2015bbe0 con 0x7f8d200aedc0 2024-05-09T04:59:58.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.784+0000 7f8d127fc640 1 -- 172.21.15.60:0/3712568414 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8d14030a20 con 0x7f8d200aedc0 2024-05-09T04:59:58.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.784+0000 7f8d127fc640 1 -- 172.21.15.60:0/3712568414 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f8d14030d10 con 0x7f8d200aedc0 2024-05-09T04:59:58.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.785+0000 7f8d27f32640 1 -- 172.21.15.60:0/3712568414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d200affb0 con 0x7f8d200aedc0 2024-05-09T04:59:58.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.787+0000 7f8d127fc640 1 -- 172.21.15.60:0/3712568414 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f8d200affb0 con 0x7f8d200aedc0 2024-05-09T04:59:58.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.850+0000 7f8d27f32640 1 -- 172.21.15.60:0/3712568414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8d20051040 con 0x7f8d200aedc0 2024-05-09T04:59:58.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.850+0000 7f8d127fc640 1 -- 172.21.15.60:0/3712568414 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f8d14047460 con 0x7f8d200aedc0 2024-05-09T04:59:58.851 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T04:59:58.851 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":10,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T04:59:58.591160+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T04:59:58.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.851+0000 7f8d27f32640 1 -- 172.21.15.60:0/3712568414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d200aedc0 msgr2=0x7f8d200ad0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:58.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.851+0000 7f8d27f32640 1 --2- 172.21.15.60:0/3712568414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d200aedc0 0x7f8d200ad0f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8d14000c00 tx=0x7f8d140046c0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.851+0000 7f8d27f32640 1 -- 172.21.15.60:0/3712568414 shutdown_connections 2024-05-09T04:59:58.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.851+0000 7f8d27f32640 1 --2- 172.21.15.60:0/3712568414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8d2015f890 0x7f8d2015c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.851+0000 7f8d27f32640 1 --2- 172.21.15.60:0/3712568414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d200b4950 0x7f8d200cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.851+0000 7f8d27f32640 1 --2- 172.21.15.60:0/3712568414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d200aedc0 0x7f8d200ad0f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:58.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.851+0000 7f8d27f32640 1 -- 172.21.15.60:0/3712568414 >> 172.21.15.60:0/3712568414 conn(0x7f8d20074780 msgr2=0x7f8d2015d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:58.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.851+0000 7f8d27f32640 1 -- 172.21.15.60:0/3712568414 shutdown_connections 2024-05-09T04:59:58.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T04:59:58.851+0000 7f8d27f32640 1 -- 172.21.15.60:0/3712568414 wait complete. 2024-05-09T04:59:58.863 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-05-09T04:59:58.863 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-09T04:59:58.900 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T04:59:58.899+0000 7f4fbd7ff040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T04:59:58.904 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:58.904+0000 7f74e271c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T04:59:59.046 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T04:59:59.046+0000 7f4fbd7ff040 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-09T04:59:59.046 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:59.046+0000 7f74e271c040 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-09T04:59:59.219 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-05-09T04:59:59.218+0000 7f01f7a1a640 -1 Falling back to public interface 2024-05-09T04:59:59.219 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-05-09T04:59:59.218+0000 7f6458216640 -1 Falling back to public interface 2024-05-09T04:59:59.219 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-05-09T04:59:59.218+0000 7f6456b99640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2024-05-09T04:59:59.219 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-05-09T04:59:59.218+0000 7f6456b99640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6801/0: (98) Address already in use 2024-05-09T04:59:59.229 INFO:tasks.ceph.osd.7.smithi114.stderr:2024-05-09T04:59:59.229+0000 7fdfb8c16640 -1 Falling back to public interface 2024-05-09T04:59:59.230 INFO:tasks.ceph.osd.4.smithi114.stderr:2024-05-09T04:59:59.229+0000 7f26f081b640 -1 Falling back to public interface 2024-05-09T04:59:59.234 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-05-09T04:59:59.234+0000 7f415abd8640 -1 Falling back to public interface 2024-05-09T04:59:59.238 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-05-09T04:59:59.238+0000 7fac72c16640 -1 Falling back to public interface 2024-05-09T04:59:59.240 INFO:tasks.ceph.osd.5.smithi114.stderr:2024-05-09T04:59:59.239+0000 7f7b2ec25640 -1 Falling back to public interface 2024-05-09T04:59:59.243 INFO:tasks.ceph.osd.6.smithi114.stderr:2024-05-09T04:59:59.242+0000 7f2c56168640 -1 Falling back to public interface 2024-05-09T04:59:59.312 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:59.312+0000 7f74e271c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T04:59:59.320 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T04:59:59.320+0000 7f4fbd7ff040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T04:59:59.364 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:59.364+0000 7f74e271c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T04:59:59.375 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T04:59:59.375+0000 7f4fbd7ff040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T04:59:59.482 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:59.482+0000 7f74e271c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T04:59:59.499 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T04:59:59.498+0000 7f4fbd7ff040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T04:59:59.668 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T04:59:59.762 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.762+0000 7fe285d4e640 1 Processor -- start 2024-05-09T04:59:59.763 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.762+0000 7fe285d4e640 1 -- start start 2024-05-09T04:59:59.764 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.763+0000 7fe285d4e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe28015f470 0x7fe280071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:59.764 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.763+0000 7fe285d4e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe2800ccad0 0x7fe2800cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:59.764 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.763+0000 7fe285d4e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2800d1150 con 0x7fe2800d13e0 2024-05-09T04:59:59.764 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.763+0000 7fe285d4e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fe2800d2ca0 con 0x7fe2800ccad0 2024-05-09T04:59:59.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.763+0000 7fe285d4e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe2800d29c0 con 0x7fe28015f470 2024-05-09T04:59:59.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.763+0000 7fe27effd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe2800d13e0 0x7fe2800cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46422/0 (socket says 172.21.15.60:46422) 2024-05-09T04:59:59.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.763+0000 7fe27effd640 1 -- 172.21.15.60:0/3530926247 learned_addr learned my addr 172.21.15.60:0/3530926247 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T04:59:59.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.763+0000 7fe27ffff640 1 --2- 172.21.15.60:0/3530926247 >> v2:172.21.15.60:3300/0 conn(0x7fe2800ccad0 0x7fe2800cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:59.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.763+0000 7fe27f7fe640 1 --2- 172.21.15.60:0/3530926247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe28015f470 0x7fe280071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:59.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.763+0000 7fe27e7fc640 1 -- 172.21.15.60:0/3530926247 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3032485997 0 0) 0x7fe2800d1150 con 0x7fe2800d13e0 2024-05-09T04:59:59.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.763+0000 7fe27e7fc640 1 -- 172.21.15.60:0/3530926247 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe268003520 con 0x7fe2800d13e0 2024-05-09T04:59:59.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.764+0000 7fe27ffff640 1 -- 172.21.15.60:0/3530926247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe28015f470 msgr2=0x7fe280071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T04:59:59.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.764+0000 7fe27ffff640 1 --2- 172.21.15.60:0/3530926247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe28015f470 0x7fe280071730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:59.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.764+0000 7fe27ffff640 1 -- 172.21.15.60:0/3530926247 >> v1:172.21.15.60:6789/0 conn(0x7fe2800d13e0 legacy=0x7fe2800cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:59.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.764+0000 7fe27e7fc640 1 -- 172.21.15.60:0/3530926247 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 157711421 0 0) 0x7fe268003520 con 0x7fe2800d13e0 2024-05-09T04:59:59.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.764+0000 7fe27ffff640 1 -- 172.21.15.60:0/3530926247 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe280162f40 con 0x7fe2800ccad0 2024-05-09T04:59:59.766 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.765+0000 7fe27ffff640 1 --2- 172.21.15.60:0/3530926247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2800ccad0 0x7fe2800cfb50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe278009d90 tx=0x7fe27802a4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae588c67b57dc97 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:59.767 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.765+0000 7fe27e7fc640 1 -- 172.21.15.60:0/3530926247 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2740016f0 con 0x7fe2800ccad0 2024-05-09T04:59:59.767 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.765+0000 7fe27e7fc640 1 -- 172.21.15.60:0/3530926247 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe278007540 con 0x7fe2800ccad0 2024-05-09T04:59:59.767 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.765+0000 7fe27e7fc640 1 -- 172.21.15.60:0/3530926247 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe278007910 con 0x7fe2800ccad0 2024-05-09T04:59:59.767 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.765+0000 7fe285d4e640 1 -- 172.21.15.60:0/3530926247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2800ccad0 msgr2=0x7fe2800cfb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:59.767 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.765+0000 7fe285d4e640 1 --2- 172.21.15.60:0/3530926247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2800ccad0 0x7fe2800cfb50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe278009d90 tx=0x7fe27802a4d0 comp rx=0 tx=0).stop 2024-05-09T04:59:59.767 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.766+0000 7fe285d4e640 1 -- 172.21.15.60:0/3530926247 shutdown_connections 2024-05-09T04:59:59.767 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.766+0000 7fe285d4e640 1 --2- 172.21.15.60:0/3530926247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2800ccad0 0x7fe2800cfb50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:59.767 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.766+0000 7fe285d4e640 1 --2- 172.21.15.60:0/3530926247 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe28015f470 0x7fe280071730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:59.767 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.766+0000 7fe285d4e640 1 -- 172.21.15.60:0/3530926247 >> 172.21.15.60:0/3530926247 conn(0x7fe2800d1d70 msgr2=0x7fe2800d2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:59.767 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.766+0000 7fe285d4e640 1 -- 172.21.15.60:0/3530926247 shutdown_connections 2024-05-09T04:59:59.767 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.766+0000 7fe285d4e640 1 -- 172.21.15.60:0/3530926247 wait complete. 2024-05-09T04:59:59.770 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.767+0000 7fe285d4e640 1 Processor -- start 2024-05-09T04:59:59.770 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.767+0000 7fe285d4e640 1 -- start start 2024-05-09T04:59:59.770 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.768+0000 7fe285d4e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2800ccad0 0x7fe2800c8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:59.770 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.768+0000 7fe285d4e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2800d13e0 0x7fe28014d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:59.770 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.768+0000 7fe285d4e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe28014daf0 0x7fe2800c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T04:59:59.770 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.768+0000 7fe285d4e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe280163a20 con 0x7fe28014daf0 2024-05-09T04:59:59.770 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.768+0000 7fe285d4e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe280163500 con 0x7fe2800ccad0 2024-05-09T04:59:59.770 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.768+0000 7fe285d4e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe280163ba0 con 0x7fe2800d13e0 2024-05-09T04:59:59.770 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.768+0000 7fe27ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe28014daf0 0x7fe2800c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:59.770 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.768+0000 7fe27ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe28014daf0 0x7fe2800c97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33694/0 (socket says 172.21.15.60:33694) 2024-05-09T04:59:59.770 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.768+0000 7fe27ffff640 1 -- 172.21.15.60:0/3268485039 learned_addr learned my addr 172.21.15.60:0/3268485039 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T04:59:59.770 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.768+0000 7fe27f7fe640 1 --2- 172.21.15.60:0/3268485039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2800ccad0 0x7fe2800c8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:59.770 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.768+0000 7fe27effd640 1 --2- 172.21.15.60:0/3268485039 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2800d13e0 0x7fe28014d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T04:59:59.770 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.768+0000 7fe27ffff640 1 -- 172.21.15.60:0/3268485039 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2800d13e0 msgr2=0x7fe28014d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:59.771 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.768+0000 7fe27ffff640 1 --2- 172.21.15.60:0/3268485039 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2800d13e0 0x7fe28014d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:59.771 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.768+0000 7fe27ffff640 1 -- 172.21.15.60:0/3268485039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2800ccad0 msgr2=0x7fe2800c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:59.771 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.768+0000 7fe27ffff640 1 --2- 172.21.15.60:0/3268485039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2800ccad0 0x7fe2800c8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:59.771 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.768+0000 7fe27ffff640 1 -- 172.21.15.60:0/3268485039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28014dfd0 con 0x7fe28014daf0 2024-05-09T04:59:59.771 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.769+0000 7fe27ffff640 1 --2- 172.21.15.60:0/3268485039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe28014daf0 0x7fe2800c97b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe278009eb0 tx=0x7fe278004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T04:59:59.771 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.769+0000 7fe267fff640 1 -- 172.21.15.60:0/3268485039 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe278040070 con 0x7fe28014daf0 2024-05-09T04:59:59.771 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.769+0000 7fe267fff640 1 -- 172.21.15.60:0/3268485039 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe278031bc0 con 0x7fe28014daf0 2024-05-09T04:59:59.771 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.769+0000 7fe267fff640 1 -- 172.21.15.60:0/3268485039 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe278007e90 con 0x7fe28014daf0 2024-05-09T04:59:59.771 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.769+0000 7fe285d4e640 1 -- 172.21.15.60:0/3268485039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe28014e2b0 con 0x7fe28014daf0 2024-05-09T04:59:59.771 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.769+0000 7fe285d4e640 1 -- 172.21.15.60:0/3268485039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe28014e870 con 0x7fe28014daf0 2024-05-09T04:59:59.771 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.769+0000 7fe267fff640 1 -- 172.21.15.60:0/3268485039 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe278039040 con 0x7fe28014daf0 2024-05-09T04:59:59.771 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.769+0000 7fe267fff640 1 -- 172.21.15.60:0/3268485039 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fe27803ba80 con 0x7fe28014daf0 2024-05-09T04:59:59.772 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.770+0000 7fe285d4e640 1 -- 172.21.15.60:0/3268485039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe280069dd0 con 0x7fe28014daf0 2024-05-09T04:59:59.772 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.772+0000 7fe267fff640 1 -- 172.21.15.60:0/3268485039 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fe27803bcd0 con 0x7fe28014daf0 2024-05-09T04:59:59.782 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:59.781+0000 7f74e271c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T04:59:59.811 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T04:59:59.811+0000 7f4fbd7ff040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T04:59:59.831 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:59.831+0000 7f74e271c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T04:59:59.835 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.835+0000 7fe285d4e640 1 -- 172.21.15.60:0/3268485039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe280076720 con 0x7fe28014daf0 2024-05-09T04:59:59.836 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.836+0000 7fe267fff640 1 -- 172.21.15.60:0/3268485039 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fe27803c940 con 0x7fe28014daf0 2024-05-09T04:59:59.836 INFO:teuthology.misc.health.smithi060.stdout: 2024-05-09T04:59:59.836 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T04:59:58.591160+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T04:59:59.837 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.837+0000 7fe285d4e640 1 -- 172.21.15.60:0/3268485039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe28014daf0 msgr2=0x7fe2800c97b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T04:59:59.837 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.837+0000 7fe285d4e640 1 --2- 172.21.15.60:0/3268485039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe28014daf0 0x7fe2800c97b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe278009eb0 tx=0x7fe278004a40 comp rx=0 tx=0).stop 2024-05-09T04:59:59.837 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.837+0000 7fe285d4e640 1 -- 172.21.15.60:0/3268485039 shutdown_connections 2024-05-09T04:59:59.837 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.837+0000 7fe285d4e640 1 --2- 172.21.15.60:0/3268485039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe28014daf0 0x7fe2800c97b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:59.838 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.837+0000 7fe285d4e640 1 --2- 172.21.15.60:0/3268485039 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2800d13e0 0x7fe28014d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:59.838 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.837+0000 7fe285d4e640 1 --2- 172.21.15.60:0/3268485039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2800ccad0 0x7fe2800c8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T04:59:59.838 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.837+0000 7fe285d4e640 1 -- 172.21.15.60:0/3268485039 >> 172.21.15.60:0/3268485039 conn(0x7fe2800d1d70 msgr2=0x7fe280070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T04:59:59.838 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.837+0000 7fe285d4e640 1 -- 172.21.15.60:0/3268485039 shutdown_connections 2024-05-09T04:59:59.838 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T04:59:59.837+0000 7fe285d4e640 1 -- 172.21.15.60:0/3268485039 wait complete. 2024-05-09T04:59:59.847 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-09T04:59:59.862 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T04:59:59.862+0000 7f4fbd7ff040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T04:59:59.990 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T04:59:59.990+0000 7f74e271c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T05:00:00.027 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T05:00:00.027+0000 7f4fbd7ff040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T05:00:00.129 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T05:00:00.129+0000 7f74e271c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T05:00:00.173 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T05:00:00.172+0000 7f4fbd7ff040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T05:00:00.179 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T05:00:00.179+0000 7f74e271c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T05:00:00.225 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T05:00:00.224+0000 7f4fbd7ff040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T05:00:00.250 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T05:00:00.250+0000 7f74e271c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T05:00:00.296 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T05:00:00.295+0000 7f74e271c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T05:00:00.298 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T05:00:00.298+0000 7f4fbd7ff040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T05:00:00.346 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T05:00:00.346+0000 7f4fbd7ff040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T05:00:00.779 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T05:00:00.778+0000 7f74e271c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T05:00:00.852 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T05:00:00.852+0000 7f4fbd7ff040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T05:00:00.897 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T05:00:00.896+0000 7f74e271c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T05:00:00.979 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-05-09T05:00:00.979+0000 7f4fbd7ff040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T05:00:01.915 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T05:00:01.914+0000 7f748203b640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-09T05:00:01.915 INFO:tasks.ceph.mgr.x.smithi114.stderr:2024-05-09T05:00:01.914+0000 7f748203b640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-09T05:00:02.679 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-05-09T05:00:02.679+0000 7f6458216640 -1 osd.1 0 log_to_monitors true 2024-05-09T05:00:02.718 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-05-09T05:00:02.718+0000 7f01f7a1a640 -1 osd.0 0 log_to_monitors true 2024-05-09T05:00:02.736 INFO:tasks.ceph.osd.7.smithi114.stderr:2024-05-09T05:00:02.735+0000 7fdfb8c16640 -1 osd.7 0 log_to_monitors true 2024-05-09T05:00:02.742 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-05-09T05:00:02.742+0000 7f415abd8640 -1 osd.2 0 log_to_monitors true 2024-05-09T05:00:02.746 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-05-09T05:00:02.746+0000 7fac72c16640 -1 osd.3 0 log_to_monitors true 2024-05-09T05:00:02.760 INFO:tasks.ceph.osd.6.smithi114.stderr:2024-05-09T05:00:02.759+0000 7f2c56168640 -1 osd.6 0 log_to_monitors true 2024-05-09T05:00:02.761 INFO:tasks.ceph.osd.5.smithi114.stderr:2024-05-09T05:00:02.760+0000 7f7b2ec25640 -1 osd.5 0 log_to_monitors true 2024-05-09T05:00:02.764 INFO:tasks.ceph.osd.4.smithi114.stderr:2024-05-09T05:00:02.764+0000 7f26f081b640 -1 osd.4 0 log_to_monitors true 2024-05-09T05:00:05.342 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-05-09T05:00:05.342+0000 7fac6e62a640 -1 osd.3 0 waiting for initial osdmap 2024-05-09T05:00:05.346 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-05-09T05:00:05.345+0000 7f01f33a0640 -1 osd.0 0 waiting for initial osdmap 2024-05-09T05:00:05.347 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-05-09T05:00:05.346+0000 7f4156b6e640 -1 osd.2 0 waiting for initial osdmap 2024-05-09T05:00:05.356 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-05-09T05:00:05.356+0000 7f6453b80640 -1 osd.1 0 waiting for initial osdmap 2024-05-09T05:00:05.358 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-05-09T05:00:05.358+0000 7fac6644a640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T05:00:05.363 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-05-09T05:00:05.363+0000 7f01eb1c0640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T05:00:05.377 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-05-09T05:00:05.377+0000 7f644b9a0640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T05:00:05.378 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-05-09T05:00:05.378+0000 7f414e98e640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T05:00:06.013 INFO:tasks.ceph.osd.6.smithi114.stderr:2024-05-09T05:00:06.012+0000 7f2c520fe640 -1 osd.6 0 waiting for initial osdmap 2024-05-09T05:00:06.015 INFO:tasks.ceph.osd.5.smithi114.stderr:2024-05-09T05:00:06.014+0000 7f7b2a667640 -1 osd.5 0 waiting for initial osdmap 2024-05-09T05:00:06.031 INFO:tasks.ceph.osd.6.smithi114.stderr:2024-05-09T05:00:06.031+0000 7f2c49f1e640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T05:00:06.034 INFO:tasks.ceph.osd.5.smithi114.stderr:2024-05-09T05:00:06.034+0000 7f7b22487640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T05:00:06.039 INFO:tasks.ceph.osd.4.smithi114.stderr:2024-05-09T05:00:06.039+0000 7f26ec207640 -1 osd.4 0 waiting for initial osdmap 2024-05-09T05:00:06.050 INFO:tasks.ceph.osd.4.smithi114.stderr:2024-05-09T05:00:06.050+0000 7f26e4027640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T05:00:06.069 INFO:tasks.ceph.osd.7.smithi114.stderr:2024-05-09T05:00:06.069+0000 7fdfb462a640 -1 osd.7 0 waiting for initial osdmap 2024-05-09T05:00:06.089 INFO:tasks.ceph.osd.7.smithi114.stderr:2024-05-09T05:00:06.089+0000 7fdfac44a640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T05:00:06.653 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T05:00:06.761 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.760+0000 7f0d5092b640 1 Processor -- start 2024-05-09T05:00:06.761 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.760+0000 7f0d5092b640 1 -- start start 2024-05-09T05:00:06.762 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.761+0000 7f0d5092b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0d4c0d1e30 0x7f0d4c0cfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:06.762 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.761+0000 7f0d5092b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d4c0d0fb0 con 0x7f0d4c15fa70 2024-05-09T05:00:06.762 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.761+0000 7f0d5092b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d4c0b7fd0 con 0x7f0d4c15f690 2024-05-09T05:00:06.763 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.761+0000 7f0d5092b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f0d4c0b82b0 con 0x7f0d4c0d1e30 2024-05-09T05:00:06.763 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.761+0000 7f0d49d9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0d4c0d1e30 0x7f0d4c0cfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:06.763 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.761+0000 7f0d4ad9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0d4c15fa70 0x7f0d4c0b8ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46426/0 (socket says 172.21.15.60:46426) 2024-05-09T05:00:06.763 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.761+0000 7f0d49d9b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0d4c0d1e30 0x7f0d4c0cfcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33736/0 (socket says 172.21.15.60:33736) 2024-05-09T05:00:06.763 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.761+0000 7f0d4ad9d640 1 -- 172.21.15.60:0/2708470103 learned_addr learned my addr 172.21.15.60:0/2708470103 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:06.763 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.762+0000 7f0d49d9b640 1 -- 172.21.15.60:0/2708470103 >> v1:172.21.15.114:6789/0 conn(0x7f0d4c15f690 legacy=0x7f0d4c0d1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:06.763 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.762+0000 7f0d49d9b640 1 -- 172.21.15.60:0/2708470103 >> v1:172.21.15.60:6789/0 conn(0x7f0d4c15fa70 legacy=0x7f0d4c0b8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:06.763 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.762+0000 7f0d49d9b640 1 -- 172.21.15.60:0/2708470103 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d4c0b8590 con 0x7f0d4c0d1e30 2024-05-09T05:00:06.764 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.762+0000 7f0d49d9b640 1 --2- 172.21.15.60:0/2708470103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c0d1e30 0x7f0d4c0cfcb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0d40009d90 tx=0x7f0d4002a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=231a4ff59c8664c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:06.764 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.763+0000 7f0d4959a640 1 -- 172.21.15.60:0/2708470103 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d40037070 con 0x7f0d4c0d1e30 2024-05-09T05:00:06.764 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.763+0000 7f0d4959a640 1 -- 172.21.15.60:0/2708470103 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d40007740 con 0x7f0d4c0d1e30 2024-05-09T05:00:06.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.763+0000 7f0d4959a640 1 -- 172.21.15.60:0/2708470103 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d40007b10 con 0x7f0d4c0d1e30 2024-05-09T05:00:06.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.763+0000 7f0d5092b640 1 -- 172.21.15.60:0/2708470103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c0d1e30 msgr2=0x7f0d4c0cfcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:06.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.763+0000 7f0d5092b640 1 --2- 172.21.15.60:0/2708470103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c0d1e30 0x7f0d4c0cfcb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0d40009d90 tx=0x7f0d4002a780 comp rx=0 tx=0).stop 2024-05-09T05:00:06.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.763+0000 7f0d5092b640 1 -- 172.21.15.60:0/2708470103 shutdown_connections 2024-05-09T05:00:06.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.763+0000 7f0d5092b640 1 --2- 172.21.15.60:0/2708470103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c0d1e30 0x7f0d4c0cfcb0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:06.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.763+0000 7f0d5092b640 1 -- 172.21.15.60:0/2708470103 >> 172.21.15.60:0/2708470103 conn(0x7f0d4c0d50e0 msgr2=0x7f0d4c0d54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:06.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.763+0000 7f0d5092b640 1 -- 172.21.15.60:0/2708470103 shutdown_connections 2024-05-09T05:00:06.765 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.763+0000 7f0d5092b640 1 -- 172.21.15.60:0/2708470103 wait complete. 2024-05-09T05:00:06.768 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.765+0000 7f0d5092b640 1 Processor -- start 2024-05-09T05:00:06.768 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.765+0000 7f0d5092b640 1 -- start start 2024-05-09T05:00:06.768 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.766+0000 7f0d5092b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c0d1e30 0x7f0d4c1f6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:06.768 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.766+0000 7f0d5092b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d4c15f690 0x7f0d4c158820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:06.768 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.766+0000 7f0d5092b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d4c15fa70 0x7f0d4c0ba5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:06.768 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.766+0000 7f0d5092b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4c0d2440 con 0x7f0d4c0d1e30 2024-05-09T05:00:06.768 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.766+0000 7f0d5092b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4c0d22c0 con 0x7f0d4c15fa70 2024-05-09T05:00:06.768 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.766+0000 7f0d5092b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f0d4c0d5c00 con 0x7f0d4c15f690 2024-05-09T05:00:06.769 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.766+0000 7f0d4a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c0d1e30 0x7f0d4c1f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:06.769 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.766+0000 7f0d4a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c0d1e30 0x7f0d4c1f6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33750/0 (socket says 172.21.15.60:33750) 2024-05-09T05:00:06.769 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.766+0000 7f0d4a59c640 1 -- 172.21.15.60:0/3563518824 learned_addr learned my addr 172.21.15.60:0/3563518824 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:06.769 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.766+0000 7f0d4ad9d640 1 --2- 172.21.15.60:0/3563518824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d4c15fa70 0x7f0d4c0ba5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:06.769 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.766+0000 7f0d49d9b640 1 --2- 172.21.15.60:0/3563518824 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d4c15f690 0x7f0d4c158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:06.769 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.766+0000 7f0d4a59c640 1 -- 172.21.15.60:0/3563518824 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d4c15f690 msgr2=0x7f0d4c158820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:06.769 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.766+0000 7f0d4a59c640 1 --2- 172.21.15.60:0/3563518824 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d4c15f690 0x7f0d4c158820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:06.769 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.766+0000 7f0d4a59c640 1 -- 172.21.15.60:0/3563518824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d4c15fa70 msgr2=0x7f0d4c0ba5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:06.769 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.766+0000 7f0d4a59c640 1 --2- 172.21.15.60:0/3563518824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d4c15fa70 0x7f0d4c0ba5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:06.769 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.766+0000 7f0d4a59c640 1 -- 172.21.15.60:0/3563518824 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d4c159070 con 0x7f0d4c0d1e30 2024-05-09T05:00:06.769 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.766+0000 7f0d4a59c640 1 --2- 172.21.15.60:0/3563518824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c0d1e30 0x7f0d4c1f6310 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c00c450 tx=0x7f0d3c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:06.769 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.767+0000 7f0d3affd640 1 -- 172.21.15.60:0/3563518824 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c017070 con 0x7f0d4c0d1e30 2024-05-09T05:00:06.770 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.767+0000 7f0d3affd640 1 -- 172.21.15.60:0/3563518824 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d3c0052d0 con 0x7f0d4c0d1e30 2024-05-09T05:00:06.770 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.767+0000 7f0d3affd640 1 -- 172.21.15.60:0/3563518824 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c003830 con 0x7f0d4c0d1e30 2024-05-09T05:00:06.770 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.767+0000 7f0d5092b640 1 -- 172.21.15.60:0/3563518824 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d4c1592e0 con 0x7f0d4c0d1e30 2024-05-09T05:00:06.770 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.767+0000 7f0d5092b640 1 -- 172.21.15.60:0/3563518824 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d4c1598a0 con 0x7f0d4c0d1e30 2024-05-09T05:00:06.771 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.768+0000 7f0d5092b640 1 -- 172.21.15.60:0/3563518824 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d4c069dd0 con 0x7f0d4c0d1e30 2024-05-09T05:00:06.771 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.769+0000 7f0d3affd640 1 -- 172.21.15.60:0/3563518824 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) ==== 98145+0+0 (secure 0 0 0) 0x7f0d3c01a070 con 0x7f0d4c0d1e30 2024-05-09T05:00:06.771 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.769+0000 7f0d3affd640 1 --2- 172.21.15.60:0/3563518824 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0d2c076240 0x7f0d2c078700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:06.771 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.769+0000 7f0d3affd640 1 -- 172.21.15.60:0/3563518824 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 4471+0+0 (secure 0 0 0) 0x7f0d3c096290 con 0x7f0d4c0d1e30 2024-05-09T05:00:06.771 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.769+0000 7f0d49d9b640 1 --2- 172.21.15.60:0/3563518824 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0d2c076240 0x7f0d2c078700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:06.771 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.770+0000 7f0d49d9b640 1 --2- 172.21.15.60:0/3563518824 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0d2c076240 0x7f0d2c078700 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0d40034000 tx=0x7f0d4002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:06.774 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.774+0000 7f0d3affd640 1 -- 172.21.15.60:0/3563518824 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0d3c05fbe0 con 0x7f0d4c0d1e30 2024-05-09T05:00:06.967 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.966+0000 7f0d5092b640 1 -- 172.21.15.60:0/3563518824 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0d4c0d5560 con 0x7f0d4c0d1e30 2024-05-09T05:00:06.968 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.967+0000 7f0d3affd640 1 -- 172.21.15.60:0/3563518824 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+10885 (secure 0 0 0) 0x7f0d3c063ee0 con 0x7f0d4c0d1e30 2024-05-09T05:00:06.968 INFO:teuthology.misc.health.smithi060.stdout: 2024-05-09T05:00:06.968 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":13,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:00:06.360267+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:00:06.970 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.969+0000 7f0d5092b640 1 -- 172.21.15.60:0/3563518824 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0d2c076240 msgr2=0x7f0d2c078700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:06.970 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.969+0000 7f0d5092b640 1 --2- 172.21.15.60:0/3563518824 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0d2c076240 0x7f0d2c078700 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0d40034000 tx=0x7f0d4002e000 comp rx=0 tx=0).stop 2024-05-09T05:00:06.970 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.969+0000 7f0d5092b640 1 -- 172.21.15.60:0/3563518824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c0d1e30 msgr2=0x7f0d4c1f6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:06.971 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.969+0000 7f0d5092b640 1 --2- 172.21.15.60:0/3563518824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c0d1e30 0x7f0d4c1f6310 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c00c450 tx=0x7f0d3c00c910 comp rx=0 tx=0).stop 2024-05-09T05:00:06.971 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.969+0000 7f0d5092b640 1 -- 172.21.15.60:0/3563518824 shutdown_connections 2024-05-09T05:00:06.971 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.969+0000 7f0d5092b640 1 --2- 172.21.15.60:0/3563518824 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0d2c076240 0x7f0d2c078700 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:06.971 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.969+0000 7f0d5092b640 1 --2- 172.21.15.60:0/3563518824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d4c15fa70 0x7f0d4c0ba5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:06.971 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.969+0000 7f0d5092b640 1 --2- 172.21.15.60:0/3563518824 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d4c15f690 0x7f0d4c158820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:06.971 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.969+0000 7f0d5092b640 1 --2- 172.21.15.60:0/3563518824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c0d1e30 0x7f0d4c1f6310 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:06.971 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.970+0000 7f0d5092b640 1 -- 172.21.15.60:0/3563518824 >> 172.21.15.60:0/3563518824 conn(0x7f0d4c0d50e0 msgr2=0x7f0d4c0b8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:06.971 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.970+0000 7f0d5092b640 1 -- 172.21.15.60:0/3563518824 shutdown_connections 2024-05-09T05:00:06.971 INFO:teuthology.misc.health.smithi060.stderr:2024-05-09T05:00:06.970+0000 7f0d5092b640 1 -- 172.21.15.60:0/3563518824 wait complete. 2024-05-09T05:00:06.980 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-05-09T05:00:06.980 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-05-09T05:00:06.980 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-09T05:00:06.980 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-09T05:00:06.983 INFO:tasks.ceph.mds.a:Started 2024-05-09T05:00:06.983 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-09T05:00:06.983 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-09T05:00:06.985 INFO:tasks.ceph.mds.c:Started 2024-05-09T05:00:06.985 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-09T05:00:06.985 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-09T05:00:06.988 INFO:tasks.ceph.mds.b:Started 2024-05-09T05:00:06.988 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-09T05:00:06.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-09T05:00:06.990 INFO:tasks.ceph.mds.d:Started 2024-05-09T05:00:06.991 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-05-09T05:00:06.991 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:00:07.062 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2024-05-09T05:00:07.068 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2024-05-09T05:00:07.075 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2024-05-09T05:00:07.082 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2024-05-09T05:00:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.101+0000 7f7c4fc2a640 1 Processor -- start 2024-05-09T05:00:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.102+0000 7f7c4fc2a640 1 -- start start 2024-05-09T05:00:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.102+0000 7f7c4fc2a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7c4815d2b0 0x7f7c480abe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.102+0000 7f7c4fc2a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c480aa380 0x7f7c480b4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.102+0000 7f7c4fc2a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c4806c840 con 0x7f7c480b4e90 2024-05-09T05:00:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.102+0000 7f7c4fc2a640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f7c4806cb20 con 0x7f7c4815d2b0 2024-05-09T05:00:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.102+0000 7f7c4fc2a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7c4806ce00 con 0x7f7c480aa380 2024-05-09T05:00:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.103+0000 7f7c4d9c6640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7c4815d2b0 0x7f7c480abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.103+0000 7f7c4d9c6640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7c4815d2b0 0x7f7c480abe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.60:38204/0 (socket says 172.21.15.60:38204) 2024-05-09T05:00:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.103+0000 7f7c4d9c6640 1 -- 172.21.15.60:0/290914281 learned_addr learned my addr 172.21.15.60:0/290914281 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.103+0000 7f7c4e1c7640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7c480b4e90 0x7f7c4815d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.60:34314/0 (socket says 172.21.15.60:34314) 2024-05-09T05:00:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.103+0000 7f7c4d1c5640 1 --2- 172.21.15.60:0/290914281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c480aa380 0x7f7c480b4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.103+0000 7f7c4d9c6640 1 -- 172.21.15.60:0/290914281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c480aa380 msgr2=0x7f7c480b4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.103+0000 7f7c4d9c6640 1 --2- 172.21.15.60:0/290914281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c480aa380 0x7f7c480b4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.103+0000 7f7c4d9c6640 1 -- 172.21.15.60:0/290914281 >> v1:172.21.15.114:6789/0 conn(0x7f7c480b4e90 legacy=0x7f7c4815d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.103+0000 7f7c4d9c6640 1 -- 172.21.15.60:0/290914281 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4806d0e0 con 0x7f7c4815d2b0 2024-05-09T05:00:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.104+0000 7f7c4d9c6640 1 --2- 172.21.15.60:0/290914281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c4815d2b0 0x7f7c480abe00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009870 tx=0x7f7c4402a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66dd5799ffd0d246 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.104+0000 7f7c4c9c4640 1 -- 172.21.15.60:0/290914281 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44037070 con 0x7f7c4815d2b0 2024-05-09T05:00:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.104+0000 7f7c4c9c4640 1 -- 172.21.15.60:0/290914281 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c44032440 con 0x7f7c4815d2b0 2024-05-09T05:00:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.104+0000 7f7c4c9c4640 1 -- 172.21.15.60:0/290914281 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44032810 con 0x7f7c4815d2b0 2024-05-09T05:00:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.104+0000 7f7c4fc2a640 1 -- 172.21.15.60:0/290914281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c4815d2b0 msgr2=0x7f7c480abe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.104+0000 7f7c4fc2a640 1 --2- 172.21.15.60:0/290914281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c4815d2b0 0x7f7c480abe00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009870 tx=0x7f7c4402a1c0 comp rx=0 tx=0).stop 2024-05-09T05:00:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.104+0000 7f7c4fc2a640 1 -- 172.21.15.60:0/290914281 shutdown_connections 2024-05-09T05:00:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.104+0000 7f7c4fc2a640 1 --2- 172.21.15.60:0/290914281 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c480aa380 0x7f7c480b4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.104+0000 7f7c4fc2a640 1 --2- 172.21.15.60:0/290914281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c4815d2b0 0x7f7c480abe00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.104+0000 7f7c4fc2a640 1 -- 172.21.15.60:0/290914281 >> 172.21.15.60:0/290914281 conn(0x7f7c48074780 msgr2=0x7f7c48073240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:07.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.104+0000 7f7c4fc2a640 1 -- 172.21.15.60:0/290914281 shutdown_connections 2024-05-09T05:00:07.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.104+0000 7f7c4fc2a640 1 -- 172.21.15.60:0/290914281 wait complete. 2024-05-09T05:00:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.106+0000 7f7c4fc2a640 1 Processor -- start 2024-05-09T05:00:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.106+0000 7f7c4fc2a640 1 -- start start 2024-05-09T05:00:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.107+0000 7f7c4fc2a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c480aa380 0x7f7c481496e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.107+0000 7f7c4fc2a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c480b4e90 0x7f7c481586c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.107+0000 7f7c4fc2a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c4815d2b0 0x7f7c480b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.107+0000 7f7c4fc2a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7c480b0aa0 con 0x7f7c4815d2b0 2024-05-09T05:00:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.107+0000 7f7c4fc2a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7c480ac340 con 0x7f7c480b4e90 2024-05-09T05:00:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.107+0000 7f7c4fc2a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7c480b0c20 con 0x7f7c480aa380 2024-05-09T05:00:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.107+0000 7f7c4e1c7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c4815d2b0 0x7f7c480b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.107+0000 7f7c4e1c7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c4815d2b0 0x7f7c480b3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:33806/0 (socket says 172.21.15.60:33806) 2024-05-09T05:00:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.107+0000 7f7c4e1c7640 1 -- 172.21.15.60:0/1009506747 learned_addr learned my addr 172.21.15.60:0/1009506747 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.107+0000 7f7c4d9c6640 1 --2- 172.21.15.60:0/1009506747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c480aa380 0x7f7c481496e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.107+0000 7f7c4d1c5640 1 --2- 172.21.15.60:0/1009506747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c480b4e90 0x7f7c481586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.107+0000 7f7c4e1c7640 1 -- 172.21.15.60:0/1009506747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c480aa380 msgr2=0x7f7c481496e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.107+0000 7f7c4e1c7640 1 --2- 172.21.15.60:0/1009506747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c480aa380 0x7f7c481496e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.107+0000 7f7c4e1c7640 1 -- 172.21.15.60:0/1009506747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c480b4e90 msgr2=0x7f7c481586c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.107+0000 7f7c4e1c7640 1 --2- 172.21.15.60:0/1009506747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c480b4e90 0x7f7c481586c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.107+0000 7f7c4e1c7640 1 -- 172.21.15.60:0/1009506747 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c48156c20 con 0x7f7c4815d2b0 2024-05-09T05:00:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.107+0000 7f7c4e1c7640 1 --2- 172.21.15.60:0/1009506747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c4815d2b0 0x7f7c480b3260 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000c770 tx=0x7f7c4000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.107+0000 7f7c3e7fc640 1 -- 172.21.15.60:0/1009506747 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40017070 con 0x7f7c4815d2b0 2024-05-09T05:00:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.107+0000 7f7c3e7fc640 1 -- 172.21.15.60:0/1009506747 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c4000a640 con 0x7f7c4815d2b0 2024-05-09T05:00:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.108+0000 7f7c4fc2a640 1 -- 172.21.15.60:0/1009506747 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c48156f00 con 0x7f7c4815d2b0 2024-05-09T05:00:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.108+0000 7f7c3e7fc640 1 -- 172.21.15.60:0/1009506747 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4000aa30 con 0x7f7c4815d2b0 2024-05-09T05:00:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.108+0000 7f7c4fc2a640 1 -- 172.21.15.60:0/1009506747 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c481574c0 con 0x7f7c4815d2b0 2024-05-09T05:00:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.109+0000 7f7c4fc2a640 1 -- 172.21.15.60:0/1009506747 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c48051040 con 0x7f7c4815d2b0 2024-05-09T05:00:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.109+0000 7f7c3e7fc640 1 -- 172.21.15.60:0/1009506747 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) ==== 98145+0+0 (secure 0 0 0) 0x7f7c4001a070 con 0x7f7c4815d2b0 2024-05-09T05:00:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.110+0000 7f7c3e7fc640 1 --2- 172.21.15.60:0/1009506747 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7c2c076240 0x7f7c2c078700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.110+0000 7f7c3e7fc640 1 -- 172.21.15.60:0/1009506747 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 4471+0+0 (secure 0 0 0) 0x7f7c400965b0 con 0x7f7c4815d2b0 2024-05-09T05:00:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.110+0000 7f7c4d9c6640 1 --2- 172.21.15.60:0/1009506747 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7c2c076240 0x7f7c2c078700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.111+0000 7f7c4d9c6640 1 --2- 172.21.15.60:0/1009506747 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7c2c076240 0x7f7c2c078700 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7c44034000 tx=0x7f7c4402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:07.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.115+0000 7f7c3e7fc640 1 -- 172.21.15.60:0/1009506747 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7c4005fe70 con 0x7f7c4815d2b0 2024-05-09T05:00:07.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.343+0000 7f7c4fc2a640 1 -- 172.21.15.60:0/1009506747 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c480694a0 con 0x7f7c4815d2b0 2024-05-09T05:00:07.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.344+0000 7f7c3e7fc640 1 -- 172.21.15.60:0/1009506747 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1392 (secure 0 0 0) 0x7f7c40064170 con 0x7f7c4815d2b0 2024-05-09T05:00:07.344 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:07.344 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1285340937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1285340937},{"type":"v1","addr":"172.21.15.60:6833","nonce":1285340937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-09T05:00:07.345 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 2 2024-05-09T05:00:07.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.345+0000 7f7c4fc2a640 1 -- 172.21.15.60:0/1009506747 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7c2c076240 msgr2=0x7f7c2c078700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:07.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.345+0000 7f7c4fc2a640 1 --2- 172.21.15.60:0/1009506747 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7c2c076240 0x7f7c2c078700 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7c44034000 tx=0x7f7c4402e000 comp rx=0 tx=0).stop 2024-05-09T05:00:07.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.345+0000 7f7c4fc2a640 1 -- 172.21.15.60:0/1009506747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c4815d2b0 msgr2=0x7f7c480b3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:07.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.345+0000 7f7c4fc2a640 1 --2- 172.21.15.60:0/1009506747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c4815d2b0 0x7f7c480b3260 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000c770 tx=0x7f7c4000cc30 comp rx=0 tx=0).stop 2024-05-09T05:00:07.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.346+0000 7f7c4fc2a640 1 -- 172.21.15.60:0/1009506747 shutdown_connections 2024-05-09T05:00:07.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.346+0000 7f7c4fc2a640 1 --2- 172.21.15.60:0/1009506747 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7c2c076240 0x7f7c2c078700 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:07.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.346+0000 7f7c4fc2a640 1 --2- 172.21.15.60:0/1009506747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c4815d2b0 0x7f7c480b3260 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:07.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.346+0000 7f7c4fc2a640 1 --2- 172.21.15.60:0/1009506747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c480b4e90 0x7f7c481586c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:07.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.346+0000 7f7c4fc2a640 1 --2- 172.21.15.60:0/1009506747 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7c480aa380 0x7f7c481496e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:07.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.346+0000 7f7c4fc2a640 1 -- 172.21.15.60:0/1009506747 >> 172.21.15.60:0/1009506747 conn(0x7f7c48074780 msgr2=0x7f7c480b0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:07.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.346+0000 7f7c4fc2a640 1 -- 172.21.15.60:0/1009506747 shutdown_connections 2024-05-09T05:00:07.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:07.346+0000 7f7c4fc2a640 1 -- 172.21.15.60:0/1009506747 wait complete. 2024-05-09T05:00:09.358 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:00:09.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.463+0000 7f7b54e6d640 1 Processor -- start 2024-05-09T05:00:09.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.463+0000 7f7b54e6d640 1 -- start start 2024-05-09T05:00:09.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.464+0000 7f7b54e6d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7b5015f300 0x7f7b501639f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:09.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.464+0000 7f7b54e6d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b501606f0 0x7f7b5006ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.464+0000 7f7b54e6d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b5006ff60 con 0x7f7b5006f9f0 2024-05-09T05:00:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.464+0000 7f7b54e6d640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f7b50070240 con 0x7f7b5015f300 2024-05-09T05:00:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.464+0000 7f7b54e6d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7b50070520 con 0x7f7b501606f0 2024-05-09T05:00:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.464+0000 7f7b4e59c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7b5015f300 0x7f7b501639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.464+0000 7f7b4e59c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7b5015f300 0x7f7b501639f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.60:55660/0 (socket says 172.21.15.60:55660) 2024-05-09T05:00:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.464+0000 7f7b45d9b640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7b5006f9f0 0x7f7b500d33f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.60:50980/0 (socket says 172.21.15.60:50980) 2024-05-09T05:00:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.464+0000 7f7b4e59c640 1 -- 172.21.15.60:0/4117943256 learned_addr learned my addr 172.21.15.60:0/4117943256 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.464+0000 7f7b4ed9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b501606f0 0x7f7b5006ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.464+0000 7f7b4e59c640 1 -- 172.21.15.60:0/4117943256 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b501606f0 msgr2=0x7f7b5006ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.464+0000 7f7b4e59c640 1 --2- 172.21.15.60:0/4117943256 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b501606f0 0x7f7b5006ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.464+0000 7f7b4e59c640 1 -- 172.21.15.60:0/4117943256 >> v1:172.21.15.114:6789/0 conn(0x7f7b5006f9f0 legacy=0x7f7b500d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.464+0000 7f7b4e59c640 1 -- 172.21.15.60:0/4117943256 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b5006e740 con 0x7f7b5015f300 2024-05-09T05:00:09.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.465+0000 7f7b4e59c640 1 --2- 172.21.15.60:0/4117943256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b5015f300 0x7f7b501639f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7b40009870 tx=0x7f7b4002a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccae3a48523354e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:09.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.465+0000 7f7b4dd9b640 1 -- 172.21.15.60:0/4117943256 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b40037070 con 0x7f7b5015f300 2024-05-09T05:00:09.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.465+0000 7f7b4dd9b640 1 -- 172.21.15.60:0/4117943256 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b40032440 con 0x7f7b5015f300 2024-05-09T05:00:09.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.465+0000 7f7b4dd9b640 1 -- 172.21.15.60:0/4117943256 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b40032810 con 0x7f7b5015f300 2024-05-09T05:00:09.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.465+0000 7f7b54e6d640 1 -- 172.21.15.60:0/4117943256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b5015f300 msgr2=0x7f7b501639f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:09.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.465+0000 7f7b54e6d640 1 --2- 172.21.15.60:0/4117943256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b5015f300 0x7f7b501639f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7b40009870 tx=0x7f7b4002a270 comp rx=0 tx=0).stop 2024-05-09T05:00:09.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.466+0000 7f7b54e6d640 1 -- 172.21.15.60:0/4117943256 shutdown_connections 2024-05-09T05:00:09.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.466+0000 7f7b54e6d640 1 --2- 172.21.15.60:0/4117943256 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b501606f0 0x7f7b5006ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:09.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.466+0000 7f7b54e6d640 1 --2- 172.21.15.60:0/4117943256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b5015f300 0x7f7b501639f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:09.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.466+0000 7f7b54e6d640 1 -- 172.21.15.60:0/4117943256 >> 172.21.15.60:0/4117943256 conn(0x7f7b5009b850 msgr2=0x7f7b5009bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:09.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.466+0000 7f7b54e6d640 1 -- 172.21.15.60:0/4117943256 shutdown_connections 2024-05-09T05:00:09.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.466+0000 7f7b54e6d640 1 -- 172.21.15.60:0/4117943256 wait complete. 2024-05-09T05:00:09.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.467+0000 7f7b54e6d640 1 Processor -- start 2024-05-09T05:00:09.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.467+0000 7f7b54e6d640 1 -- start start 2024-05-09T05:00:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.468+0000 7f7b54e6d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b5006f9f0 0x7f7b5015e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.468+0000 7f7b54e6d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b5015f300 0x7f7b501f5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.468+0000 7f7b54e6d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b501606f0 0x7f7b500af7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.468+0000 7f7b54e6d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7b500d3b00 con 0x7f7b5006f9f0 2024-05-09T05:00:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.468+0000 7f7b54e6d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7b50163f30 con 0x7f7b5015f300 2024-05-09T05:00:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.468+0000 7f7b54e6d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7b500d3c80 con 0x7f7b501606f0 2024-05-09T05:00:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.468+0000 7f7b4e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b5006f9f0 0x7f7b5015e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.468+0000 7f7b4e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b5006f9f0 0x7f7b5015e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40416/0 (socket says 172.21.15.60:40416) 2024-05-09T05:00:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.468+0000 7f7b4e59c640 1 -- 172.21.15.60:0/2999313891 learned_addr learned my addr 172.21.15.60:0/2999313891 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.468+0000 7f7b45d9b640 1 --2- 172.21.15.60:0/2999313891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b5015f300 0x7f7b501f5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.468+0000 7f7b4ed9d640 1 --2- 172.21.15.60:0/2999313891 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b501606f0 0x7f7b500af7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.468+0000 7f7b4e59c640 1 -- 172.21.15.60:0/2999313891 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b501606f0 msgr2=0x7f7b500af7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.468+0000 7f7b4e59c640 1 --2- 172.21.15.60:0/2999313891 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b501606f0 0x7f7b500af7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.468+0000 7f7b4e59c640 1 -- 172.21.15.60:0/2999313891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b5015f300 msgr2=0x7f7b501f5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.468+0000 7f7b4e59c640 1 --2- 172.21.15.60:0/2999313891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b5015f300 0x7f7b501f5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.468+0000 7f7b4e59c640 1 -- 172.21.15.60:0/2999313891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b501f60b0 con 0x7f7b5006f9f0 2024-05-09T05:00:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.468+0000 7f7b45d9b640 1 --2- 172.21.15.60:0/2999313891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b5015f300 0x7f7b501f5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:00:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.469+0000 7f7b4e59c640 1 --2- 172.21.15.60:0/2999313891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b5006f9f0 0x7f7b5015e340 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7b40000c00 tx=0x7f7b400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.469+0000 7f7b477fe640 1 -- 172.21.15.60:0/2999313891 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b40037040 con 0x7f7b5006f9f0 2024-05-09T05:00:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.469+0000 7f7b477fe640 1 -- 172.21.15.60:0/2999313891 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b40042070 con 0x7f7b5006f9f0 2024-05-09T05:00:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.469+0000 7f7b54e6d640 1 -- 172.21.15.60:0/2999313891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b500ae4b0 con 0x7f7b5006f9f0 2024-05-09T05:00:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.469+0000 7f7b477fe640 1 -- 172.21.15.60:0/2999313891 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b40030880 con 0x7f7b5006f9f0 2024-05-09T05:00:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.469+0000 7f7b54e6d640 1 -- 172.21.15.60:0/2999313891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b500aea30 con 0x7f7b5006f9f0 2024-05-09T05:00:09.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.470+0000 7f7b54e6d640 1 -- 172.21.15.60:0/2999313891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b500af420 con 0x7f7b5006f9f0 2024-05-09T05:00:09.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.470+0000 7f7b477fe640 1 -- 172.21.15.60:0/2999313891 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) ==== 98145+0+0 (secure 0 0 0) 0x7f7b40030a20 con 0x7f7b5006f9f0 2024-05-09T05:00:09.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.471+0000 7f7b477fe640 1 --2- 172.21.15.60:0/2999313891 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7b30076240 0x7f7b30078700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:09.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.471+0000 7f7b477fe640 1 -- 172.21.15.60:0/2999313891 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f7b400b7920 con 0x7f7b5006f9f0 2024-05-09T05:00:09.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.471+0000 7f7b45d9b640 1 --2- 172.21.15.60:0/2999313891 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7b30076240 0x7f7b30078700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:09.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.472+0000 7f7b45d9b640 1 --2- 172.21.15.60:0/2999313891 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7b30076240 0x7f7b30078700 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7b500d4e90 tx=0x7f7b38000bc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:09.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.476+0000 7f7b477fe640 1 -- 172.21.15.60:0/2999313891 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7b501f60b0 con 0x7f7b5006f9f0 2024-05-09T05:00:09.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.706+0000 7f7b54e6d640 1 -- 172.21.15.60:0/2999313891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b50051040 con 0x7f7b5006f9f0 2024-05-09T05:00:09.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.707+0000 7f7b477fe640 1 -- 172.21.15.60:0/2999313891 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3870 (secure 0 0 0) 0x7f7b40080fe0 con 0x7f7b5006f9f0 2024-05-09T05:00:09.707 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:09.707 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1285340937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1285340937},{"type":"v1","addr":"172.21.15.60:6833","nonce":1285340937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2599193390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2599193390},{"type":"v1","addr":"172.21.15.114:6835","nonce":2599193390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3432609307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3432609307},{"type":"v1","addr":"172.21.15.114:6837","nonce":3432609307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4261,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/2135423314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2135423314},{"type":"v1","addr":"172.21.15.60:6835","nonce":2135423314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-09T05:00:09.708 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 3 2024-05-09T05:00:09.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.709+0000 7f7b54e6d640 1 -- 172.21.15.60:0/2999313891 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7b30076240 msgr2=0x7f7b30078700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:09.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.709+0000 7f7b54e6d640 1 --2- 172.21.15.60:0/2999313891 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7b30076240 0x7f7b30078700 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7b500d4e90 tx=0x7f7b38000bc0 comp rx=0 tx=0).stop 2024-05-09T05:00:09.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.709+0000 7f7b54e6d640 1 -- 172.21.15.60:0/2999313891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b5006f9f0 msgr2=0x7f7b5015e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:09.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.709+0000 7f7b54e6d640 1 --2- 172.21.15.60:0/2999313891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b5006f9f0 0x7f7b5015e340 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7b40000c00 tx=0x7f7b400046c0 comp rx=0 tx=0).stop 2024-05-09T05:00:09.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.709+0000 7f7b54e6d640 1 -- 172.21.15.60:0/2999313891 shutdown_connections 2024-05-09T05:00:09.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.709+0000 7f7b54e6d640 1 --2- 172.21.15.60:0/2999313891 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7b30076240 0x7f7b30078700 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:09.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.709+0000 7f7b54e6d640 1 --2- 172.21.15.60:0/2999313891 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7b501606f0 0x7f7b500af7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:09.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.709+0000 7f7b54e6d640 1 --2- 172.21.15.60:0/2999313891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b5015f300 0x7f7b501f5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:09.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.709+0000 7f7b54e6d640 1 --2- 172.21.15.60:0/2999313891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b5006f9f0 0x7f7b5015e340 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:09.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.709+0000 7f7b54e6d640 1 -- 172.21.15.60:0/2999313891 >> 172.21.15.60:0/2999313891 conn(0x7f7b5009b850 msgr2=0x7f7b50160af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:09.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.709+0000 7f7b54e6d640 1 -- 172.21.15.60:0/2999313891 shutdown_connections 2024-05-09T05:00:09.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.709+0000 7f7b54e6d640 1 -- 172.21.15.60:0/2999313891 wait complete. 2024-05-09T05:00:09.720 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-09T05:00:09.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.825+0000 7f3f12d80640 1 Processor -- start 2024-05-09T05:00:09.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.825+0000 7f3f12d80640 1 -- start start 2024-05-09T05:00:09.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f12d80640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f3f0c070580 0x7f3f0c09c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:09.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f12d80640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f0c09e3e0 con 0x7f3f0c15f2e0 2024-05-09T05:00:09.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f12d80640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f0c09e6c0 con 0x7f3f0c15f6c0 2024-05-09T05:00:09.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f12d80640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f3f0c09ed10 con 0x7f3f0c070580 2024-05-09T05:00:09.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f10b1c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3f0c15f2e0 0x7f3f0c09f1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43506/0 (socket says 172.21.15.60:43506) 2024-05-09T05:00:09.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f10b1c640 1 -- 172.21.15.60:0/4264082862 learned_addr learned my addr 172.21.15.60:0/4264082862 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:09.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f0b7fe640 1 -- 172.21.15.60:0/4264082862 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1255295089 0 0) 0x7f3f0c09e3e0 con 0x7f3f0c15f2e0 2024-05-09T05:00:09.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f0b7fe640 1 -- 172.21.15.60:0/4264082862 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3efc003520 con 0x7f3f0c15f2e0 2024-05-09T05:00:09.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f0bfff640 1 --2- 172.21.15.60:0/4264082862 >> v2:172.21.15.114:3300/0 conn(0x7f3f0c070580 0x7f3f0c09c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:09.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f0b7fe640 1 -- 172.21.15.60:0/4264082862 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 381534557 0 0) 0x7f3efc003520 con 0x7f3f0c15f2e0 2024-05-09T05:00:09.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f0b7fe640 1 -- 172.21.15.60:0/4264082862 >> v2:172.21.15.114:3300/0 conn(0x7f3f0c070580 msgr2=0x7f3f0c09c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:09.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f0b7fe640 1 --2- 172.21.15.60:0/4264082862 >> v2:172.21.15.114:3300/0 conn(0x7f3f0c070580 0x7f3f0c09c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:09.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f0b7fe640 1 -- 172.21.15.60:0/4264082862 >> v1:172.21.15.114:6789/0 conn(0x7f3f0c15f6c0 legacy=0x7f3f0c0b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:09.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f0b7fe640 1 -- 172.21.15.60:0/4264082862 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f0c09e9a0 con 0x7f3f0c15f2e0 2024-05-09T05:00:09.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f0b7fe640 1 -- 172.21.15.60:0/4264082862 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f3f000016f0 con 0x7f3f0c15f2e0 2024-05-09T05:00:09.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f0b7fe640 1 -- 172.21.15.60:0/4264082862 >> v1:172.21.15.60:6789/0 conn(0x7f3f0c15f2e0 legacy=0x7f3f0c09f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:09.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f0b7fe640 1 --2- 172.21.15.60:0/4264082862 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3efc003a50 0x7f3efc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:09.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f0b7fe640 1 --2- 172.21.15.60:0/4264082862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3efc004800 0x7f3efc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:09.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f0b7fe640 1 --2- 172.21.15.60:0/4264082862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3efc0052e0 0x7f3efc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:09.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f0b7fe640 1 -- 172.21.15.60:0/4264082862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c09ed10 con 0x7f3efc004800 2024-05-09T05:00:09.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f0b7fe640 1 -- 172.21.15.60:0/4264082862 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc00c1b0 con 0x7f3efc0052e0 2024-05-09T05:00:09.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f0b7fe640 1 -- 172.21.15.60:0/4264082862 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f3efc00c3b0 con 0x7f3efc003a50 2024-05-09T05:00:09.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.826+0000 7f3f0bfff640 1 --2- 172.21.15.60:0/4264082862 >> v2:172.21.15.114:3300/0 conn(0x7f3f0c070580 0x7f3f0c09c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:09.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.827+0000 7f3f0bfff640 1 --2- 172.21.15.60:0/4264082862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3efc004800 0x7f3efc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:09.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.827+0000 7f3f10b1c640 1 --2- 172.21.15.60:0/4264082862 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3efc003a50 0x7f3efc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:09.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.827+0000 7f3f1131d640 1 --2- 172.21.15.60:0/4264082862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3efc0052e0 0x7f3efc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:09.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.827+0000 7f3f1131d640 1 -- 172.21.15.60:0/4264082862 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3efc003a50 msgr2=0x7f3efc003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:09.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.827+0000 7f3f1131d640 1 --2- 172.21.15.60:0/4264082862 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3efc003a50 0x7f3efc003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:09.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.827+0000 7f3f1131d640 1 -- 172.21.15.60:0/4264082862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3efc004800 msgr2=0x7f3efc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:09.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.827+0000 7f3f1131d640 1 --2- 172.21.15.60:0/4264082862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3efc004800 0x7f3efc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:09.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.827+0000 7f3f1131d640 1 -- 172.21.15.60:0/4264082862 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3efc003520 con 0x7f3efc0052e0 2024-05-09T05:00:09.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.827+0000 7f3f0bfff640 1 --2- 172.21.15.60:0/4264082862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3efc004800 0x7f3efc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:00:09.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.827+0000 7f3f10b1c640 1 --2- 172.21.15.60:0/4264082862 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3efc003a50 0x7f3efc003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:00:09.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.827+0000 7f3f1131d640 1 -- 172.21.15.60:0/4264082862 reap_dead start 2024-05-09T05:00:09.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.828+0000 7f3f1131d640 1 --2- 172.21.15.60:0/4264082862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3efc0052e0 0x7f3efc00b9c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3f04001e60 tx=0x7f3efc005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4abc61528042ac9b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:09.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.828+0000 7f3f0b7fe640 1 -- 172.21.15.60:0/4264082862 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f00007d30 con 0x7f3efc0052e0 2024-05-09T05:00:09.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.828+0000 7f3f0b7fe640 1 -- 172.21.15.60:0/4264082862 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f0402ae00 con 0x7f3efc0052e0 2024-05-09T05:00:09.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.828+0000 7f3f0b7fe640 1 -- 172.21.15.60:0/4264082862 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef800f040 con 0x7f3efc0052e0 2024-05-09T05:00:09.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.828+0000 7f3f12d80640 1 -- 172.21.15.60:0/4264082862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3efc0052e0 msgr2=0x7f3efc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:09.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.828+0000 7f3f12d80640 1 --2- 172.21.15.60:0/4264082862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3efc0052e0 0x7f3efc00b9c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3f04001e60 tx=0x7f3efc005830 comp rx=0 tx=0).stop 2024-05-09T05:00:09.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.828+0000 7f3f12d80640 1 -- 172.21.15.60:0/4264082862 shutdown_connections 2024-05-09T05:00:09.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.828+0000 7f3f12d80640 1 --2- 172.21.15.60:0/4264082862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3efc0052e0 0x7f3efc00b9c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:09.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.828+0000 7f3f12d80640 1 -- 172.21.15.60:0/4264082862 >> 172.21.15.60:0/4264082862 conn(0x7f3f0c0d4900 msgr2=0x7f3f0c0d4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:09.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.828+0000 7f3f12d80640 1 -- 172.21.15.60:0/4264082862 shutdown_connections 2024-05-09T05:00:09.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.828+0000 7f3f12d80640 1 -- 172.21.15.60:0/4264082862 wait complete. 2024-05-09T05:00:09.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.829+0000 7f3f12d80640 1 Processor -- start 2024-05-09T05:00:09.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.830+0000 7f3f12d80640 1 -- start start 2024-05-09T05:00:09.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.830+0000 7f3f12d80640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3efc0052e0 0x7f3f0c072850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:09.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.830+0000 7f3f12d80640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3f0c0b1450 0x7f3f0c0abe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:09.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.830+0000 7f3f12d80640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f0c0d3050 0x7f3f0c161530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:09.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.830+0000 7f3f12d80640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c163130 con 0x7f3efc0052e0 2024-05-09T05:00:09.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.830+0000 7f3f12d80640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c162fb0 con 0x7f3f0c0d3050 2024-05-09T05:00:09.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.830+0000 7f3f12d80640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f3f0c09c820 con 0x7f3f0c0b1450 2024-05-09T05:00:09.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.830+0000 7f3f10b1c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3efc0052e0 0x7f3f0c072850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:09.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.831+0000 7f3f10b1c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3efc0052e0 0x7f3f0c072850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40444/0 (socket says 172.21.15.60:40444) 2024-05-09T05:00:09.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.831+0000 7f3f10b1c640 1 -- 172.21.15.60:0/3689686693 learned_addr learned my addr 172.21.15.60:0/3689686693 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:09.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.831+0000 7f3f0bfff640 1 --2- 172.21.15.60:0/3689686693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3f0c0b1450 0x7f3f0c0abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:09.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.831+0000 7f3f1131d640 1 --2- 172.21.15.60:0/3689686693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f0c0d3050 0x7f3f0c161530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:09.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.831+0000 7f3f10b1c640 1 -- 172.21.15.60:0/3689686693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3f0c0b1450 msgr2=0x7f3f0c0abe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:09.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.831+0000 7f3f10b1c640 1 --2- 172.21.15.60:0/3689686693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3f0c0b1450 0x7f3f0c0abe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:09.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.831+0000 7f3f10b1c640 1 -- 172.21.15.60:0/3689686693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f0c0d3050 msgr2=0x7f3f0c161530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:09.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.831+0000 7f3f10b1c640 1 --2- 172.21.15.60:0/3689686693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f0c0d3050 0x7f3f0c161530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:09.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.831+0000 7f3f10b1c640 1 -- 172.21.15.60:0/3689686693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f0c0b3ce0 con 0x7f3efc0052e0 2024-05-09T05:00:09.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.831+0000 7f3f1131d640 1 --2- 172.21.15.60:0/3689686693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f0c0d3050 0x7f3f0c161530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:00:09.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.831+0000 7f3f10b1c640 1 --2- 172.21.15.60:0/3689686693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3efc0052e0 0x7f3f0c072850 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3f00007310 tx=0x7f3f000077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:09.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.831+0000 7f3f097fa640 1 -- 172.21.15.60:0/3689686693 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0000f8e0 con 0x7f3efc0052e0 2024-05-09T05:00:09.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.831+0000 7f3f097fa640 1 -- 172.21.15.60:0/3689686693 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f00018070 con 0x7f3efc0052e0 2024-05-09T05:00:09.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.831+0000 7f3f12d80640 1 -- 172.21.15.60:0/3689686693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f0c0d2290 con 0x7f3efc0052e0 2024-05-09T05:00:09.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.831+0000 7f3f097fa640 1 -- 172.21.15.60:0/3689686693 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0000fc80 con 0x7f3efc0052e0 2024-05-09T05:00:09.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.831+0000 7f3f12d80640 1 -- 172.21.15.60:0/3689686693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f0c0d2850 con 0x7f3efc0052e0 2024-05-09T05:00:09.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.832+0000 7f3f12d80640 1 -- 172.21.15.60:0/3689686693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f0c051040 con 0x7f3efc0052e0 2024-05-09T05:00:09.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.833+0000 7f3f097fa640 1 -- 172.21.15.60:0/3689686693 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3f0001d460 con 0x7f3efc0052e0 2024-05-09T05:00:09.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.833+0000 7f3f097fa640 1 --2- 172.21.15.60:0/3689686693 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f3ef00762d0 0x7f3ef0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:09.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.833+0000 7f3f097fa640 1 -- 172.21.15.60:0/3689686693 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f3f00097170 con 0x7f3efc0052e0 2024-05-09T05:00:09.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.833+0000 7f3f0bfff640 1 --2- 172.21.15.60:0/3689686693 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f3ef00762d0 0x7f3ef0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:09.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.834+0000 7f3f0bfff640 1 --2- 172.21.15.60:0/3689686693 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f3ef00762d0 0x7f3ef0078790 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8006fa0 tx=0x7f3ef8010040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:09.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:09.838+0000 7f3f097fa640 1 -- 172.21.15.60:0/3689686693 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3f000607d0 con 0x7f3efc0052e0 2024-05-09T05:00:10.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.029+0000 7f3f12d80640 1 -- 172.21.15.60:0/3689686693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f3f0c0b1830 con 0x7f3efc0052e0 2024-05-09T05:00:10.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.029+0000 7f3f097fa640 1 -- 172.21.15.60:0/3689686693 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7f3f00064ad0 con 0x7f3efc0052e0 2024-05-09T05:00:10.030 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:10.030 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2024-05-09T05:00:10.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:10.031 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2024-05-09T05:00:10.031 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2024-05-09T05:00:10.031 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:10.031 INFO:teuthology.orchestra.run.smithi060.stdout:] 2024-05-09T05:00:10.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.031+0000 7f3f12d80640 1 -- 172.21.15.60:0/3689686693 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f3ef00762d0 msgr2=0x7f3ef0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:10.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.031+0000 7f3f12d80640 1 --2- 172.21.15.60:0/3689686693 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f3ef00762d0 0x7f3ef0078790 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8006fa0 tx=0x7f3ef8010040 comp rx=0 tx=0).stop 2024-05-09T05:00:10.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.031+0000 7f3f12d80640 1 -- 172.21.15.60:0/3689686693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3efc0052e0 msgr2=0x7f3f0c072850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:10.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.031+0000 7f3f12d80640 1 --2- 172.21.15.60:0/3689686693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3efc0052e0 0x7f3f0c072850 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3f00007310 tx=0x7f3f000077d0 comp rx=0 tx=0).stop 2024-05-09T05:00:10.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.032+0000 7f3f12d80640 1 -- 172.21.15.60:0/3689686693 shutdown_connections 2024-05-09T05:00:10.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.032+0000 7f3f12d80640 1 --2- 172.21.15.60:0/3689686693 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f3ef00762d0 0x7f3ef0078790 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:10.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.032+0000 7f3f12d80640 1 --2- 172.21.15.60:0/3689686693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3f0c0d3050 0x7f3f0c161530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:10.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.032+0000 7f3f12d80640 1 --2- 172.21.15.60:0/3689686693 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3f0c0b1450 0x7f3f0c0abe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:10.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.032+0000 7f3f12d80640 1 --2- 172.21.15.60:0/3689686693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3efc0052e0 0x7f3f0c072850 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:10.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.032+0000 7f3f12d80640 1 -- 172.21.15.60:0/3689686693 >> 172.21.15.60:0/3689686693 conn(0x7f3f0c0d4900 msgr2=0x7f3f0c073930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:10.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.032+0000 7f3f12d80640 1 -- 172.21.15.60:0/3689686693 shutdown_connections 2024-05-09T05:00:10.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.032+0000 7f3f12d80640 1 -- 172.21.15.60:0/3689686693 wait complete. 2024-05-09T05:00:10.043 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-09T05:00:10.043 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-09T05:00:10.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.150+0000 7f4a941af640 1 Processor -- start 2024-05-09T05:00:10.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.150+0000 7f4a941af640 1 -- start start 2024-05-09T05:00:10.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.151+0000 7f4a941af640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4a8c15f4d0 0x7f4a8c0b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:10.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.151+0000 7f4a941af640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8c0b6b30 0x7f4a8c0d2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:10.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.151+0000 7f4a941af640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a8c0b5550 con 0x7f4a8c15f0f0 2024-05-09T05:00:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.151+0000 7f4a941af640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f4a8c0b6160 con 0x7f4a8c15f4d0 2024-05-09T05:00:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.151+0000 7f4a941af640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4a8c0b9600 con 0x7f4a8c0b6b30 2024-05-09T05:00:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.151+0000 7f4a91f4b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4a8c15f0f0 0x7f4a8c0b8ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43512/0 (socket says 172.21.15.60:43512) 2024-05-09T05:00:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.151+0000 7f4a91f4b640 1 -- 172.21.15.60:0/1924315171 learned_addr learned my addr 172.21.15.60:0/1924315171 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.151+0000 7f4a9174a640 1 --2- 172.21.15.60:0/1924315171 >> v2:172.21.15.114:3300/0 conn(0x7f4a8c15f4d0 0x7f4a8c0b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.151+0000 7f4a90f49640 1 -- 172.21.15.60:0/1924315171 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 538996256 0 0) 0x7f4a8c0b5550 con 0x7f4a8c15f0f0 2024-05-09T05:00:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.151+0000 7f4a9274c640 1 --2- 172.21.15.60:0/1924315171 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8c0b6b30 0x7f4a8c0d2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.151+0000 7f4a90f49640 1 -- 172.21.15.60:0/1924315171 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a78003520 con 0x7f4a8c15f0f0 2024-05-09T05:00:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a90f49640 1 -- 172.21.15.60:0/1924315171 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 962172399 0 0) 0x7f4a78003520 con 0x7f4a8c15f0f0 2024-05-09T05:00:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a90f49640 1 -- 172.21.15.60:0/1924315171 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8c0b6b30 msgr2=0x7f4a8c0d2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a90f49640 1 --2- 172.21.15.60:0/1924315171 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8c0b6b30 0x7f4a8c0d2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a90f49640 1 -- 172.21.15.60:0/1924315171 >> v2:172.21.15.114:3300/0 conn(0x7f4a8c15f4d0 msgr2=0x7f4a8c0b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a90f49640 1 --2- 172.21.15.60:0/1924315171 >> v2:172.21.15.114:3300/0 conn(0x7f4a8c15f4d0 0x7f4a8c0b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:10.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a90f49640 1 -- 172.21.15.60:0/1924315171 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8c0b5830 con 0x7f4a8c15f0f0 2024-05-09T05:00:10.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a90f49640 1 -- 172.21.15.60:0/1924315171 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f4a880016f0 con 0x7f4a8c15f0f0 2024-05-09T05:00:10.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a90f49640 1 -- 172.21.15.60:0/1924315171 >> v1:172.21.15.60:6789/0 conn(0x7f4a8c15f0f0 legacy=0x7f4a8c0b8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:10.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a90f49640 1 --2- 172.21.15.60:0/1924315171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a78003a00 0x7f4a78003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:10.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a90f49640 1 --2- 172.21.15.60:0/1924315171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a78004800 0x7f4a78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:10.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a90f49640 1 --2- 172.21.15.60:0/1924315171 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8c0b6b30 0x7f4a780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:10.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a90f49640 1 -- 172.21.15.60:0/1924315171 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c0b6160 con 0x7f4a78004800 2024-05-09T05:00:10.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a90f49640 1 -- 172.21.15.60:0/1924315171 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c0b9600 con 0x7f4a78003a00 2024-05-09T05:00:10.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a90f49640 1 -- 172.21.15.60:0/1924315171 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4a7800b9c0 con 0x7f4a8c0b6b30 2024-05-09T05:00:10.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a9174a640 1 --2- 172.21.15.60:0/1924315171 >> v2:172.21.15.114:3300/0 conn(0x7f4a8c15f4d0 0x7f4a8c0b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:00:10.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a91f4b640 1 --2- 172.21.15.60:0/1924315171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a78003a00 0x7f4a78003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:10.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a9274c640 1 --2- 172.21.15.60:0/1924315171 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8c0b6b30 0x7f4a780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:10.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a9174a640 1 --2- 172.21.15.60:0/1924315171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a78004800 0x7f4a78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:10.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a91f4b640 1 -- 172.21.15.60:0/1924315171 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8c0b6b30 msgr2=0x7f4a780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:10.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a91f4b640 1 --2- 172.21.15.60:0/1924315171 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8c0b6b30 0x7f4a780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:10.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a91f4b640 1 -- 172.21.15.60:0/1924315171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a78004800 msgr2=0x7f4a78004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:10.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a91f4b640 1 --2- 172.21.15.60:0/1924315171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a78004800 0x7f4a78004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:10.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.152+0000 7f4a91f4b640 1 -- 172.21.15.60:0/1924315171 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a78003520 con 0x7f4a78003a00 2024-05-09T05:00:10.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.153+0000 7f4a91f4b640 1 --2- 172.21.15.60:0/1924315171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a78003a00 0x7f4a78003e20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800c290 tx=0x7f4a78005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7d7f20ca54c6f1f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:10.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.153+0000 7f4a90f49640 1 -- 172.21.15.60:0/1924315171 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a88007d50 con 0x7f4a78003a00 2024-05-09T05:00:10.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.153+0000 7f4a90f49640 1 -- 172.21.15.60:0/1924315171 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a88005af0 con 0x7f4a78003a00 2024-05-09T05:00:10.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.153+0000 7f4a90f49640 1 -- 172.21.15.60:0/1924315171 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a88007070 con 0x7f4a78003a00 2024-05-09T05:00:10.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.153+0000 7f4a941af640 1 -- 172.21.15.60:0/1924315171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a78003a00 msgr2=0x7f4a78003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:10.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.153+0000 7f4a941af640 1 --2- 172.21.15.60:0/1924315171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a78003a00 0x7f4a78003e20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800c290 tx=0x7f4a78005830 comp rx=0 tx=0).stop 2024-05-09T05:00:10.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.153+0000 7f4a9274c640 1 -- 172.21.15.60:0/1924315171 reap_dead start 2024-05-09T05:00:10.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.154+0000 7f4a941af640 1 -- 172.21.15.60:0/1924315171 shutdown_connections 2024-05-09T05:00:10.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.154+0000 7f4a941af640 1 -- 172.21.15.60:0/1924315171 >> 172.21.15.60:0/1924315171 conn(0x7f4a8c06ebd0 msgr2=0x7f4a8c073840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:10.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.154+0000 7f4a941af640 1 -- 172.21.15.60:0/1924315171 shutdown_connections 2024-05-09T05:00:10.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.154+0000 7f4a941af640 1 -- 172.21.15.60:0/1924315171 wait complete. 2024-05-09T05:00:10.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.155+0000 7f4a941af640 1 Processor -- start 2024-05-09T05:00:10.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.155+0000 7f4a941af640 1 -- start start 2024-05-09T05:00:10.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a941af640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a78003a00 0x7f4a8c1512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:10.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a941af640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8c153910 0x7f4a8c0d4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:10.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a941af640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a8c0d4b70 0x7f4a8c15f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:10.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a941af640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c0bac60 con 0x7f4a78003a00 2024-05-09T05:00:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a941af640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c0b5df0 con 0x7f4a8c0d4b70 2024-05-09T05:00:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a941af640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4a8c0bade0 con 0x7f4a8c153910 2024-05-09T05:00:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a91f4b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a78003a00 0x7f4a8c1512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a91f4b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a78003a00 0x7f4a8c1512b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40460/0 (socket says 172.21.15.60:40460) 2024-05-09T05:00:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a91f4b640 1 -- 172.21.15.60:0/1291322743 learned_addr learned my addr 172.21.15.60:0/1291322743 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a9274c640 1 --2- 172.21.15.60:0/1291322743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a8c0d4b70 0x7f4a8c15f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a9174a640 1 --2- 172.21.15.60:0/1291322743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8c153910 0x7f4a8c0d4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a91f4b640 1 -- 172.21.15.60:0/1291322743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8c153910 msgr2=0x7f4a8c0d4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a91f4b640 1 --2- 172.21.15.60:0/1291322743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8c153910 0x7f4a8c0d4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a91f4b640 1 -- 172.21.15.60:0/1291322743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a8c0d4b70 msgr2=0x7f4a8c15f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a91f4b640 1 --2- 172.21.15.60:0/1291322743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a8c0d4b70 0x7f4a8c15f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a91f4b640 1 -- 172.21.15.60:0/1291322743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8c1f5a60 con 0x7f4a78003a00 2024-05-09T05:00:10.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a91f4b640 1 --2- 172.21.15.60:0/1291322743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a78003a00 0x7f4a8c1512b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800c380 tx=0x7f4a8c0b5720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:10.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a827fc640 1 -- 172.21.15.60:0/1291322743 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a88043070 con 0x7f4a78003a00 2024-05-09T05:00:10.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a827fc640 1 -- 172.21.15.60:0/1291322743 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a88038040 con 0x7f4a78003a00 2024-05-09T05:00:10.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a827fc640 1 -- 172.21.15.60:0/1291322743 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a88007070 con 0x7f4a78003a00 2024-05-09T05:00:10.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a941af640 1 -- 172.21.15.60:0/1291322743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a8c1f5d40 con 0x7f4a78003a00 2024-05-09T05:00:10.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.156+0000 7f4a941af640 1 -- 172.21.15.60:0/1291322743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a8c1f6300 con 0x7f4a78003a00 2024-05-09T05:00:10.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.157+0000 7f4a941af640 1 -- 172.21.15.60:0/1291322743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a8c051040 con 0x7f4a78003a00 2024-05-09T05:00:10.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.158+0000 7f4a827fc640 1 -- 172.21.15.60:0/1291322743 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4a88007210 con 0x7f4a78003a00 2024-05-09T05:00:10.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.158+0000 7f4a827fc640 1 --2- 172.21.15.60:0/1291322743 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4a700762d0 0x7f4a70078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:10.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.159+0000 7f4a827fc640 1 -- 172.21.15.60:0/1291322743 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f4a880b6d50 con 0x7f4a78003a00 2024-05-09T05:00:10.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.159+0000 7f4a9174a640 1 --2- 172.21.15.60:0/1291322743 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4a700762d0 0x7f4a70078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:10.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.159+0000 7f4a9174a640 1 --2- 172.21.15.60:0/1291322743 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4a700762d0 0x7f4a70078790 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c154f60 tx=0x7f4a7c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:10.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.163+0000 7f4a827fc640 1 -- 172.21.15.60:0/1291322743 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4a880ba030 con 0x7f4a78003a00 2024-05-09T05:00:10.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.353+0000 7f4a941af640 1 -- 172.21.15.60:0/1291322743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f4a8c0d4f50 con 0x7f4a78003a00 2024-05-09T05:00:10.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.815+0000 7f4a827fc640 1 -- 172.21.15.60:0/1291322743 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7f4a88005ac0 con 0x7f4a78003a00 2024-05-09T05:00:10.816 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2024-05-09T05:00:10.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.818+0000 7f4a941af640 1 -- 172.21.15.60:0/1291322743 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4a700762d0 msgr2=0x7f4a70078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:10.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.818+0000 7f4a941af640 1 --2- 172.21.15.60:0/1291322743 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4a700762d0 0x7f4a70078790 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c154f60 tx=0x7f4a7c009190 comp rx=0 tx=0).stop 2024-05-09T05:00:10.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.818+0000 7f4a941af640 1 -- 172.21.15.60:0/1291322743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a78003a00 msgr2=0x7f4a8c1512b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:10.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.818+0000 7f4a941af640 1 --2- 172.21.15.60:0/1291322743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a78003a00 0x7f4a8c1512b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800c380 tx=0x7f4a8c0b5720 comp rx=0 tx=0).stop 2024-05-09T05:00:10.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.818+0000 7f4a941af640 1 -- 172.21.15.60:0/1291322743 shutdown_connections 2024-05-09T05:00:10.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.818+0000 7f4a941af640 1 --2- 172.21.15.60:0/1291322743 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4a700762d0 0x7f4a70078790 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:10.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.818+0000 7f4a941af640 1 --2- 172.21.15.60:0/1291322743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a8c0d4b70 0x7f4a8c15f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:10.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.818+0000 7f4a941af640 1 --2- 172.21.15.60:0/1291322743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a8c153910 0x7f4a8c0d4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:10.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.818+0000 7f4a941af640 1 --2- 172.21.15.60:0/1291322743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a78003a00 0x7f4a8c1512b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:10.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.818+0000 7f4a941af640 1 -- 172.21.15.60:0/1291322743 >> 172.21.15.60:0/1291322743 conn(0x7f4a8c06ebd0 msgr2=0x7f4a8c071f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:10.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.819+0000 7f4a941af640 1 -- 172.21.15.60:0/1291322743 shutdown_connections 2024-05-09T05:00:10.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.819+0000 7f4a941af640 1 -- 172.21.15.60:0/1291322743 wait complete. 2024-05-09T05:00:10.833 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-05-09T05:00:10.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.949+0000 7fe5f5d2b640 1 Processor -- start 2024-05-09T05:00:10.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.949+0000 7fe5f5d2b640 1 -- start start 2024-05-09T05:00:10.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.950+0000 7fe5f5d2b640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe5f00b9d00 0x7fe5f015f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:10.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.950+0000 7fe5f5d2b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5f0163c40 con 0x7fe5f0163620 2024-05-09T05:00:10.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.950+0000 7fe5f5d2b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5f015fb90 con 0x7fe5f00ba0e0 2024-05-09T05:00:10.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.950+0000 7fe5f5d2b640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fe5f015fe70 con 0x7fe5f00b9d00 2024-05-09T05:00:10.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.950+0000 7fe5ef7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe5f00b9d00 0x7fe5f015f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:10.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.950+0000 7fe5ef7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe5f00b9d00 0x7fe5f015f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40466/0 (socket says 172.21.15.60:40466) 2024-05-09T05:00:10.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.950+0000 7fe5ef7fe640 1 -- 172.21.15.60:0/1686508421 learned_addr learned my addr 172.21.15.60:0/1686508421 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:10.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.951+0000 7fe5ef7fe640 1 -- 172.21.15.60:0/1686508421 >> v1:172.21.15.114:6789/0 conn(0x7fe5f00ba0e0 legacy=0x7fe5f0162f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:10.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.951+0000 7fe5ef7fe640 1 -- 172.21.15.60:0/1686508421 >> v1:172.21.15.60:6789/0 conn(0x7fe5f0163620 legacy=0x7fe5f00b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:10.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.951+0000 7fe5ee7fc640 1 -- 172.21.15.60:0/1686508421 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 555865347 0 0) 0x7fe5f0163c40 con 0x7fe5f0163620 2024-05-09T05:00:10.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.951+0000 7fe5ef7fe640 1 -- 172.21.15.60:0/1686508421 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f0160150 con 0x7fe5f00b9d00 2024-05-09T05:00:10.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.951+0000 7fe5ef7fe640 1 --2- 172.21.15.60:0/1686508421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5f00b9d00 0x7fe5f015f490 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0009e50 tx=0x7fe5e002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1660000b97fa8ec server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:10.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.952+0000 7fe5ee7fc640 1 -- 172.21.15.60:0/1686508421 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0036070 con 0x7fe5f00b9d00 2024-05-09T05:00:10.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.952+0000 7fe5ee7fc640 1 -- 172.21.15.60:0/1686508421 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5e0031600 con 0x7fe5f00b9d00 2024-05-09T05:00:10.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.952+0000 7fe5ee7fc640 1 -- 172.21.15.60:0/1686508421 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e00319d0 con 0x7fe5f00b9d00 2024-05-09T05:00:10.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.952+0000 7fe5f5d2b640 1 -- 172.21.15.60:0/1686508421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5f00b9d00 msgr2=0x7fe5f015f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:10.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.952+0000 7fe5f5d2b640 1 --2- 172.21.15.60:0/1686508421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5f00b9d00 0x7fe5f015f490 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0009e50 tx=0x7fe5e002a680 comp rx=0 tx=0).stop 2024-05-09T05:00:10.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.952+0000 7fe5f5d2b640 1 -- 172.21.15.60:0/1686508421 shutdown_connections 2024-05-09T05:00:10.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.952+0000 7fe5f5d2b640 1 --2- 172.21.15.60:0/1686508421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5f00b9d00 0x7fe5f015f490 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:10.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.952+0000 7fe5f5d2b640 1 -- 172.21.15.60:0/1686508421 >> 172.21.15.60:0/1686508421 conn(0x7fe5f006cd50 msgr2=0x7fe5f00b85e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:10.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.952+0000 7fe5f5d2b640 1 -- 172.21.15.60:0/1686508421 shutdown_connections 2024-05-09T05:00:10.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.952+0000 7fe5f5d2b640 1 -- 172.21.15.60:0/1686508421 wait complete. 2024-05-09T05:00:10.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.953+0000 7fe5f5d2b640 1 Processor -- start 2024-05-09T05:00:10.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.954+0000 7fe5f5d2b640 1 -- start start 2024-05-09T05:00:10.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.954+0000 7fe5f5d2b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5f00b9d00 0x7fe5f01f5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:10.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.954+0000 7fe5f5d2b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5f00ba0e0 0x7fe5f015bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:10.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.954+0000 7fe5f5d2b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5f0163620 0x7fe5f015c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:10.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.954+0000 7fe5f5d2b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f0160670 con 0x7fe5f0163620 2024-05-09T05:00:10.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.954+0000 7fe5f5d2b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f00b5bc0 con 0x7fe5f00ba0e0 2024-05-09T05:00:10.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.954+0000 7fe5f5d2b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe5f01497b0 con 0x7fe5f00b9d00 2024-05-09T05:00:10.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.954+0000 7fe5effff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5f0163620 0x7fe5f015c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:10.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.954+0000 7fe5effff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5f0163620 0x7fe5f015c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40472/0 (socket says 172.21.15.60:40472) 2024-05-09T05:00:10.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.954+0000 7fe5effff640 1 -- 172.21.15.60:0/3318086629 learned_addr learned my addr 172.21.15.60:0/3318086629 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:10.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.954+0000 7fe5ef7fe640 1 --2- 172.21.15.60:0/3318086629 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5f00b9d00 0x7fe5f01f5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:10.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.954+0000 7fe5eeffd640 1 --2- 172.21.15.60:0/3318086629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5f00ba0e0 0x7fe5f015bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:10.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.955+0000 7fe5effff640 1 -- 172.21.15.60:0/3318086629 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5f00b9d00 msgr2=0x7fe5f01f5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:10.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.955+0000 7fe5effff640 1 --2- 172.21.15.60:0/3318086629 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5f00b9d00 0x7fe5f01f5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:10.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.955+0000 7fe5effff640 1 -- 172.21.15.60:0/3318086629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5f00ba0e0 msgr2=0x7fe5f015bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:10.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.955+0000 7fe5effff640 1 --2- 172.21.15.60:0/3318086629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5f00ba0e0 0x7fe5f015bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:10.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.955+0000 7fe5effff640 1 -- 172.21.15.60:0/3318086629 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f0149a90 con 0x7fe5f0163620 2024-05-09T05:00:10.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.955+0000 7fe5effff640 1 --2- 172.21.15.60:0/3318086629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5f0163620 0x7fe5f015c320 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800c390 tx=0x7fe5e800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:10.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.955+0000 7fe5d7fff640 1 -- 172.21.15.60:0/3318086629 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8017070 con 0x7fe5f0163620 2024-05-09T05:00:10.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.955+0000 7fe5d7fff640 1 -- 172.21.15.60:0/3318086629 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5e80035f0 con 0x7fe5f0163620 2024-05-09T05:00:10.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.955+0000 7fe5f5d2b640 1 -- 172.21.15.60:0/3318086629 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5f0149d70 con 0x7fe5f0163620 2024-05-09T05:00:10.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.955+0000 7fe5d7fff640 1 -- 172.21.15.60:0/3318086629 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e80039e0 con 0x7fe5f0163620 2024-05-09T05:00:10.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.955+0000 7fe5f5d2b640 1 -- 172.21.15.60:0/3318086629 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5f014a330 con 0x7fe5f0163620 2024-05-09T05:00:10.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.956+0000 7fe5f5d2b640 1 -- 172.21.15.60:0/3318086629 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5f0072d80 con 0x7fe5f0163620 2024-05-09T05:00:10.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.957+0000 7fe5d7fff640 1 -- 172.21.15.60:0/3318086629 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe5e801a070 con 0x7fe5f0163620 2024-05-09T05:00:10.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.957+0000 7fe5d7fff640 1 --2- 172.21.15.60:0/3318086629 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe5dc0762d0 0x7fe5dc078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:10.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.958+0000 7fe5d7fff640 1 -- 172.21.15.60:0/3318086629 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7fe5e8096680 con 0x7fe5f0163620 2024-05-09T05:00:10.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.958+0000 7fe5ef7fe640 1 --2- 172.21.15.60:0/3318086629 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe5dc0762d0 0x7fe5dc078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:10.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.958+0000 7fe5ef7fe640 1 --2- 172.21.15.60:0/3318086629 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe5dc0762d0 0x7fe5dc078790 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0033000 tx=0x7fe5e0009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:10.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:10.962+0000 7fe5d7fff640 1 -- 172.21.15.60:0/3318086629 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe5f0072d80 con 0x7fe5f0163620 2024-05-09T05:00:11.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.153+0000 7fe5f5d2b640 1 -- 172.21.15.60:0/3318086629 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fe5f0051040 con 0x7fe5f0163620 2024-05-09T05:00:11.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.822+0000 7fe5d7fff640 1 -- 172.21.15.60:0/3318086629 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) ==== 171+0+0 (secure 0 0 0) 0x7fe5e805fb60 con 0x7fe5f0163620 2024-05-09T05:00:11.823 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2024-05-09T05:00:11.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.826+0000 7fe5f5d2b640 1 -- 172.21.15.60:0/3318086629 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe5dc0762d0 msgr2=0x7fe5dc078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:11.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.826+0000 7fe5f5d2b640 1 --2- 172.21.15.60:0/3318086629 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe5dc0762d0 0x7fe5dc078790 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0033000 tx=0x7fe5e0009000 comp rx=0 tx=0).stop 2024-05-09T05:00:11.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.826+0000 7fe5f5d2b640 1 -- 172.21.15.60:0/3318086629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5f0163620 msgr2=0x7fe5f015c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:11.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.826+0000 7fe5f5d2b640 1 --2- 172.21.15.60:0/3318086629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5f0163620 0x7fe5f015c320 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800c390 tx=0x7fe5e800c850 comp rx=0 tx=0).stop 2024-05-09T05:00:11.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.826+0000 7fe5f5d2b640 1 -- 172.21.15.60:0/3318086629 shutdown_connections 2024-05-09T05:00:11.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.826+0000 7fe5f5d2b640 1 --2- 172.21.15.60:0/3318086629 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe5dc0762d0 0x7fe5dc078790 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:11.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.826+0000 7fe5f5d2b640 1 --2- 172.21.15.60:0/3318086629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5f0163620 0x7fe5f015c320 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:11.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.826+0000 7fe5f5d2b640 1 --2- 172.21.15.60:0/3318086629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5f00ba0e0 0x7fe5f015bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:11.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.826+0000 7fe5f5d2b640 1 --2- 172.21.15.60:0/3318086629 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe5f00b9d00 0x7fe5f01f5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:11.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.826+0000 7fe5f5d2b640 1 -- 172.21.15.60:0/3318086629 >> 172.21.15.60:0/3318086629 conn(0x7fe5f006cd50 msgr2=0x7fe5f00b6780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:11.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.826+0000 7fe5f5d2b640 1 -- 172.21.15.60:0/3318086629 shutdown_connections 2024-05-09T05:00:11.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.827+0000 7fe5f5d2b640 1 -- 172.21.15.60:0/3318086629 wait complete. 2024-05-09T05:00:11.856 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-09T05:00:11.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.966+0000 7f612e778640 1 Processor -- start 2024-05-09T05:00:11.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.967+0000 7f612e778640 1 -- start start 2024-05-09T05:00:11.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.967+0000 7f612e778640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61280b9d00 0x7f612816a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:11.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.967+0000 7f612e778640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61280742e0 con 0x7f61280b7fe0 2024-05-09T05:00:11.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.967+0000 7f612e778640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6128163740 con 0x7f612815f3c0 2024-05-09T05:00:11.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.967+0000 7f612e778640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6128163a20 con 0x7f61280b9d00 2024-05-09T05:00:11.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.968+0000 7f61277fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f61280b7fe0 0x7f61280b95f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43524/0 (socket says 172.21.15.60:43524) 2024-05-09T05:00:11.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.968+0000 7f61277fe640 1 -- 172.21.15.60:0/1943232988 learned_addr learned my addr 172.21.15.60:0/1943232988 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:11.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.968+0000 7f612cd15640 1 --2- 172.21.15.60:0/1943232988 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61280b9d00 0x7f612816a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:11.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.968+0000 7f6126ffd640 1 -- 172.21.15.60:0/1943232988 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1953319694 0 0) 0x7f61280742e0 con 0x7f61280b7fe0 2024-05-09T05:00:11.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.968+0000 7f6126ffd640 1 -- 172.21.15.60:0/1943232988 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6110003520 con 0x7f61280b7fe0 2024-05-09T05:00:11.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.968+0000 7f6126ffd640 1 -- 172.21.15.60:0/1943232988 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2835299934 0 0) 0x7f6128163740 con 0x7f612815f3c0 2024-05-09T05:00:11.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.968+0000 7f6126ffd640 1 -- 172.21.15.60:0/1943232988 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61280742e0 con 0x7f612815f3c0 2024-05-09T05:00:11.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.968+0000 7f612cd15640 1 -- 172.21.15.60:0/1943232988 >> v1:172.21.15.114:6789/0 conn(0x7f612815f3c0 legacy=0x7f6128162e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:11.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.968+0000 7f612cd15640 1 -- 172.21.15.60:0/1943232988 >> v1:172.21.15.60:6789/0 conn(0x7f61280b7fe0 legacy=0x7f61280b95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:11.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.968+0000 7f6126ffd640 1 -- 172.21.15.60:0/1943232988 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4259958545 0 0) 0x7f6110003520 con 0x7f61280b7fe0 2024-05-09T05:00:11.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.968+0000 7f612cd15640 1 -- 172.21.15.60:0/1943232988 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6128163d00 con 0x7f61280b9d00 2024-05-09T05:00:11.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.969+0000 7f612cd15640 1 --2- 172.21.15.60:0/1943232988 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61280b9d00 0x7f612816a1d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6120009dc0 tx=0x7f612002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=279fb9a9302dd1cb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:11.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.969+0000 7f6126ffd640 1 -- 172.21.15.60:0/1943232988 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c0016f0 con 0x7f61280b9d00 2024-05-09T05:00:11.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.969+0000 7f6126ffd640 1 -- 172.21.15.60:0/1943232988 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6120007540 con 0x7f61280b9d00 2024-05-09T05:00:11.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.969+0000 7f6126ffd640 1 -- 172.21.15.60:0/1943232988 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6120007910 con 0x7f61280b9d00 2024-05-09T05:00:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.970+0000 7f612e778640 1 -- 172.21.15.60:0/1943232988 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61280b9d00 msgr2=0x7f612816a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.970+0000 7f612e778640 1 --2- 172.21.15.60:0/1943232988 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61280b9d00 0x7f612816a1d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6120009dc0 tx=0x7f612002a660 comp rx=0 tx=0).stop 2024-05-09T05:00:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.970+0000 7f612e778640 1 -- 172.21.15.60:0/1943232988 shutdown_connections 2024-05-09T05:00:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.970+0000 7f612e778640 1 --2- 172.21.15.60:0/1943232988 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61280b9d00 0x7f612816a1d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.970+0000 7f612e778640 1 -- 172.21.15.60:0/1943232988 >> 172.21.15.60:0/1943232988 conn(0x7f612806cae0 msgr2=0x7f61280d53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.970+0000 7f612e778640 1 -- 172.21.15.60:0/1943232988 shutdown_connections 2024-05-09T05:00:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.970+0000 7f612e778640 1 -- 172.21.15.60:0/1943232988 wait complete. 2024-05-09T05:00:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.971+0000 7f612e778640 1 Processor -- start 2024-05-09T05:00:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.971+0000 7f612e778640 1 -- start start 2024-05-09T05:00:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.972+0000 7f612e778640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61280b7fe0 0x7f61280b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.972+0000 7f612e778640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61280b9d00 0x7f6128159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.972+0000 7f612e778640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f612815f3c0 0x7f61281607f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.972+0000 7f612e778640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f61280d1dc0 con 0x7f612815f3c0 2024-05-09T05:00:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.972+0000 7f612e778640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6128072db0 con 0x7f61280b9d00 2024-05-09T05:00:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.972+0000 7f612e778640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f612816a710 con 0x7f61280b7fe0 2024-05-09T05:00:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.972+0000 7f612cd15640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f612815f3c0 0x7f61281607f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.972+0000 7f612cd15640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f612815f3c0 0x7f61281607f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40480/0 (socket says 172.21.15.60:40480) 2024-05-09T05:00:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.972+0000 7f612cd15640 1 -- 172.21.15.60:0/520017068 learned_addr learned my addr 172.21.15.60:0/520017068 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.972+0000 7f61277fe640 1 --2- 172.21.15.60:0/520017068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61280b9d00 0x7f6128159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.972+0000 7f6127fff640 1 --2- 172.21.15.60:0/520017068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61280b7fe0 0x7f61280b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.972+0000 7f612cd15640 1 -- 172.21.15.60:0/520017068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61280b7fe0 msgr2=0x7f61280b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.972+0000 7f612cd15640 1 --2- 172.21.15.60:0/520017068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61280b7fe0 0x7f61280b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.972+0000 7f612cd15640 1 -- 172.21.15.60:0/520017068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61280b9d00 msgr2=0x7f6128159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:11.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.972+0000 7f612cd15640 1 --2- 172.21.15.60:0/520017068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61280b9d00 0x7f6128159bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:11.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.972+0000 7f612cd15640 1 -- 172.21.15.60:0/520017068 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612806ed90 con 0x7f612815f3c0 2024-05-09T05:00:11.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.972+0000 7f612cd15640 1 --2- 172.21.15.60:0/520017068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f612815f3c0 0x7f61281607f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6120009ee0 tx=0x7f6120004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:11.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.973+0000 7f6124ff9640 1 -- 172.21.15.60:0/520017068 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6120040070 con 0x7f612815f3c0 2024-05-09T05:00:11.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.973+0000 7f6124ff9640 1 -- 172.21.15.60:0/520017068 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6120031bc0 con 0x7f612815f3c0 2024-05-09T05:00:11.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.973+0000 7f612e778640 1 -- 172.21.15.60:0/520017068 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f612806f070 con 0x7f612815f3c0 2024-05-09T05:00:11.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.973+0000 7f6124ff9640 1 -- 172.21.15.60:0/520017068 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6120009450 con 0x7f612815f3c0 2024-05-09T05:00:11.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.973+0000 7f612e778640 1 -- 172.21.15.60:0/520017068 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6128157a90 con 0x7f612815f3c0 2024-05-09T05:00:11.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.974+0000 7f612e778640 1 -- 172.21.15.60:0/520017068 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6128051040 con 0x7f612815f3c0 2024-05-09T05:00:11.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.975+0000 7f6124ff9640 1 -- 172.21.15.60:0/520017068 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6120039040 con 0x7f612815f3c0 2024-05-09T05:00:11.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.975+0000 7f6124ff9640 1 --2- 172.21.15.60:0/520017068 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f61140762d0 0x7f6114078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:11.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.975+0000 7f6124ff9640 1 -- 172.21.15.60:0/520017068 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7f61200b9190 con 0x7f612815f3c0 2024-05-09T05:00:11.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.975+0000 7f6127fff640 1 --2- 172.21.15.60:0/520017068 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f61140762d0 0x7f6114078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:11.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.976+0000 7f6127fff640 1 --2- 172.21.15.60:0/520017068 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f61140762d0 0x7f6114078790 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6118001120 tx=0x7f6118009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:11.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:11.979+0000 7f6124ff9640 1 -- 172.21.15.60:0/520017068 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6120053020 con 0x7f612815f3c0 2024-05-09T05:00:12.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.207+0000 7f612e778640 1 -- 172.21.15.60:0/520017068 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f612815ea20 con 0x7f612815f3c0 2024-05-09T05:00:12.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.824+0000 7f6124ff9640 1 -- 172.21.15.60:0/520017068 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f61200825e0 con 0x7f612815f3c0 2024-05-09T05:00:12.824 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 2 and data pool 3 2024-05-09T05:00:12.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.830+0000 7f612e778640 1 -- 172.21.15.60:0/520017068 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f61140762d0 msgr2=0x7f6114078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:12.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.830+0000 7f612e778640 1 --2- 172.21.15.60:0/520017068 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f61140762d0 0x7f6114078790 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6118001120 tx=0x7f6118009040 comp rx=0 tx=0).stop 2024-05-09T05:00:12.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.830+0000 7f612e778640 1 -- 172.21.15.60:0/520017068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f612815f3c0 msgr2=0x7f61281607f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:12.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.830+0000 7f612e778640 1 --2- 172.21.15.60:0/520017068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f612815f3c0 0x7f61281607f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6120009ee0 tx=0x7f6120004a40 comp rx=0 tx=0).stop 2024-05-09T05:00:12.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.832+0000 7f612e778640 1 -- 172.21.15.60:0/520017068 shutdown_connections 2024-05-09T05:00:12.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.832+0000 7f612e778640 1 --2- 172.21.15.60:0/520017068 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f61140762d0 0x7f6114078790 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:12.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.832+0000 7f612e778640 1 --2- 172.21.15.60:0/520017068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f612815f3c0 0x7f61281607f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:12.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.832+0000 7f612e778640 1 --2- 172.21.15.60:0/520017068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61280b9d00 0x7f6128159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:12.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.832+0000 7f612e778640 1 --2- 172.21.15.60:0/520017068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f61280b7fe0 0x7f61280b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:12.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.832+0000 7f612e778640 1 -- 172.21.15.60:0/520017068 >> 172.21.15.60:0/520017068 conn(0x7f612806cae0 msgr2=0x7f61280b5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:12.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.840+0000 7f612e778640 1 -- 172.21.15.60:0/520017068 shutdown_connections 2024-05-09T05:00:12.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.842+0000 7f612e778640 1 -- 172.21.15.60:0/520017068 wait complete. 2024-05-09T05:00:12.867 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-05-09T05:00:12.868 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-05-09T05:00:12.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.979+0000 7f30967e4640 1 Processor -- start 2024-05-09T05:00:12.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.979+0000 7f30967e4640 1 -- start start 2024-05-09T05:00:12.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.980+0000 7f30967e4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3090162e70 0x7f3090163300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:12.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.980+0000 7f30967e4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30900d31b0 con 0x7f3090148120 2024-05-09T05:00:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.980+0000 7f30967e4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30900d3490 con 0x7f30900b5e50 2024-05-09T05:00:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.980+0000 7f30967e4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f30900b5650 con 0x7f3090162e70 2024-05-09T05:00:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.980+0000 7f308ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3090148120 0x7f30900d2760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43538/0 (socket says 172.21.15.60:43538) 2024-05-09T05:00:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.980+0000 7f308ffff640 1 -- 172.21.15.60:0/3594972369 learned_addr learned my addr 172.21.15.60:0/3594972369 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.980+0000 7f308f7fe640 1 --2- 172.21.15.60:0/3594972369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3090162e70 0x7f3090163300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.981+0000 7f308effd640 1 -- 172.21.15.60:0/3594972369 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3090423216 0 0) 0x7f30900d31b0 con 0x7f3090148120 2024-05-09T05:00:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.981+0000 7f308effd640 1 -- 172.21.15.60:0/3594972369 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3078003520 con 0x7f3090148120 2024-05-09T05:00:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.981+0000 7f308effd640 1 -- 172.21.15.60:0/3594972369 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2370846442 0 0) 0x7f30900d3490 con 0x7f30900b5e50 2024-05-09T05:00:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.981+0000 7f308effd640 1 -- 172.21.15.60:0/3594972369 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30900d31b0 con 0x7f30900b5e50 2024-05-09T05:00:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.981+0000 7f308f7fe640 1 -- 172.21.15.60:0/3594972369 >> v1:172.21.15.114:6789/0 conn(0x7f30900b5e50 legacy=0x7f30900b6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.981+0000 7f308effd640 1 -- 172.21.15.60:0/3594972369 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2582959721 0 0) 0x7f3078003520 con 0x7f3090148120 2024-05-09T05:00:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.981+0000 7f308f7fe640 1 -- 172.21.15.60:0/3594972369 >> v1:172.21.15.60:6789/0 conn(0x7f3090148120 legacy=0x7f30900d2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:12.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.981+0000 7f308f7fe640 1 -- 172.21.15.60:0/3594972369 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30900b5930 con 0x7f3090162e70 2024-05-09T05:00:12.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.982+0000 7f308f7fe640 1 --2- 172.21.15.60:0/3594972369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3090162e70 0x7f3090163300 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f30840097d0 tx=0x7f308402a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e8ab6d6cfab4d51 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:12.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.982+0000 7f308effd640 1 -- 172.21.15.60:0/3594972369 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3084037070 con 0x7f3090162e70 2024-05-09T05:00:12.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.982+0000 7f308effd640 1 -- 172.21.15.60:0/3594972369 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f308402ac30 con 0x7f3090162e70 2024-05-09T05:00:12.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.982+0000 7f308effd640 1 -- 172.21.15.60:0/3594972369 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30800016f0 con 0x7f3090162e70 2024-05-09T05:00:12.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.982+0000 7f30967e4640 1 -- 172.21.15.60:0/3594972369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3090162e70 msgr2=0x7f3090163300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:12.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.982+0000 7f30967e4640 1 --2- 172.21.15.60:0/3594972369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3090162e70 0x7f3090163300 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f30840097d0 tx=0x7f308402a1a0 comp rx=0 tx=0).stop 2024-05-09T05:00:12.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.982+0000 7f30967e4640 1 -- 172.21.15.60:0/3594972369 shutdown_connections 2024-05-09T05:00:12.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.982+0000 7f30967e4640 1 --2- 172.21.15.60:0/3594972369 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3090162e70 0x7f3090163300 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:12.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.982+0000 7f30967e4640 1 -- 172.21.15.60:0/3594972369 >> 172.21.15.60:0/3594972369 conn(0x7f309006f640 msgr2=0x7f3090073840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:12.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.982+0000 7f30967e4640 1 -- 172.21.15.60:0/3594972369 shutdown_connections 2024-05-09T05:00:12.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.982+0000 7f30967e4640 1 -- 172.21.15.60:0/3594972369 wait complete. 2024-05-09T05:00:12.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.984+0000 7f30967e4640 1 Processor -- start 2024-05-09T05:00:12.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.984+0000 7f30967e4640 1 -- start start 2024-05-09T05:00:12.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.984+0000 7f30967e4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30900b5e50 0x7f3090160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:12.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.984+0000 7f30967e4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3090148120 0x7f30901611f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:12.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.984+0000 7f30967e4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3090162e70 0x7f30900b3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:12.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.985+0000 7f30967e4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3090163b30 con 0x7f3090162e70 2024-05-09T05:00:12.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.985+0000 7f30967e4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f30900d2e70 con 0x7f30900b5e50 2024-05-09T05:00:12.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.985+0000 7f30967e4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f3090163cb0 con 0x7f3090148120 2024-05-09T05:00:12.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.985+0000 7f3094d81640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3090162e70 0x7f30900b3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:12.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.985+0000 7f3094d81640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3090162e70 0x7f30900b3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40490/0 (socket says 172.21.15.60:40490) 2024-05-09T05:00:12.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.985+0000 7f3094d81640 1 -- 172.21.15.60:0/1298011404 learned_addr learned my addr 172.21.15.60:0/1298011404 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:12.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.985+0000 7f308f7fe640 1 --2- 172.21.15.60:0/1298011404 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3090148120 0x7f30901611f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:12.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.985+0000 7f308ffff640 1 --2- 172.21.15.60:0/1298011404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30900b5e50 0x7f3090160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:12.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.985+0000 7f3094d81640 1 -- 172.21.15.60:0/1298011404 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3090148120 msgr2=0x7f30901611f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:12.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.985+0000 7f3094d81640 1 --2- 172.21.15.60:0/1298011404 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3090148120 0x7f30901611f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:12.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.985+0000 7f3094d81640 1 -- 172.21.15.60:0/1298011404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30900b5e50 msgr2=0x7f3090160cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:12.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.985+0000 7f3094d81640 1 --2- 172.21.15.60:0/1298011404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30900b5e50 0x7f3090160cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:12.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.985+0000 7f3094d81640 1 -- 172.21.15.60:0/1298011404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3090143bc0 con 0x7f3090162e70 2024-05-09T05:00:12.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.985+0000 7f3094d81640 1 --2- 172.21.15.60:0/1298011404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3090162e70 0x7f30900b3a20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f308800c780 tx=0x7f308800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:12.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.985+0000 7f308cff9640 1 -- 172.21.15.60:0/1298011404 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3088003b70 con 0x7f3090162e70 2024-05-09T05:00:12.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.985+0000 7f308cff9640 1 -- 172.21.15.60:0/1298011404 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3088003d10 con 0x7f3090162e70 2024-05-09T05:00:12.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.985+0000 7f308cff9640 1 -- 172.21.15.60:0/1298011404 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f308800aa30 con 0x7f3090162e70 2024-05-09T05:00:12.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.985+0000 7f30967e4640 1 -- 172.21.15.60:0/1298011404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30901618c0 con 0x7f3090162e70 2024-05-09T05:00:12.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.986+0000 7f30967e4640 1 -- 172.21.15.60:0/1298011404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3090161e80 con 0x7f3090162e70 2024-05-09T05:00:12.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.986+0000 7f30967e4640 1 -- 172.21.15.60:0/1298011404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3090051040 con 0x7f3090162e70 2024-05-09T05:00:12.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.987+0000 7f308cff9640 1 -- 172.21.15.60:0/1298011404 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f308800abd0 con 0x7f3090162e70 2024-05-09T05:00:12.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.987+0000 7f308cff9640 1 --2- 172.21.15.60:0/1298011404 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f307c0762d0 0x7f307c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:12.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.988+0000 7f308cff9640 1 -- 172.21.15.60:0/1298011404 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f3088097370 con 0x7f3090162e70 2024-05-09T05:00:12.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.988+0000 7f308ffff640 1 --2- 172.21.15.60:0/1298011404 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f307c0762d0 0x7f307c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:12.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.988+0000 7f308ffff640 1 --2- 172.21.15.60:0/1298011404 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f307c0762d0 0x7f307c078790 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3080000c00 tx=0x7f30800016e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:12.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:12.992+0000 7f308cff9640 1 -- 172.21.15.60:0/1298011404 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3088060710 con 0x7f3090162e70 2024-05-09T05:00:13.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.190+0000 7f30967e4640 1 -- 172.21.15.60:0/1298011404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f3090162be0 con 0x7f3090162e70 2024-05-09T05:00:13.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.837+0000 7f308cff9640 1 -- 172.21.15.60:0/1298011404 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) ==== 157+0+0 (secure 0 0 0) 0x7f3088064a10 con 0x7f3090162e70 2024-05-09T05:00:13.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.841+0000 7f30967e4640 1 -- 172.21.15.60:0/1298011404 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f307c0762d0 msgr2=0x7f307c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:13.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.841+0000 7f30967e4640 1 --2- 172.21.15.60:0/1298011404 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f307c0762d0 0x7f307c078790 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3080000c00 tx=0x7f30800016e0 comp rx=0 tx=0).stop 2024-05-09T05:00:13.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.841+0000 7f30967e4640 1 -- 172.21.15.60:0/1298011404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3090162e70 msgr2=0x7f30900b3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:13.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.841+0000 7f30967e4640 1 --2- 172.21.15.60:0/1298011404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3090162e70 0x7f30900b3a20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f308800c780 tx=0x7f308800cc40 comp rx=0 tx=0).stop 2024-05-09T05:00:13.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.841+0000 7f30967e4640 1 -- 172.21.15.60:0/1298011404 shutdown_connections 2024-05-09T05:00:13.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.841+0000 7f30967e4640 1 --2- 172.21.15.60:0/1298011404 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f307c0762d0 0x7f307c078790 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:13.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.841+0000 7f30967e4640 1 --2- 172.21.15.60:0/1298011404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3090162e70 0x7f30900b3a20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:13.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.841+0000 7f30967e4640 1 --2- 172.21.15.60:0/1298011404 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3090148120 0x7f30901611f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:13.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.841+0000 7f30967e4640 1 --2- 172.21.15.60:0/1298011404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30900b5e50 0x7f3090160cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:13.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.841+0000 7f30967e4640 1 -- 172.21.15.60:0/1298011404 >> 172.21.15.60:0/1298011404 conn(0x7f309006f640 msgr2=0x7f30900b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:13.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.841+0000 7f30967e4640 1 -- 172.21.15.60:0/1298011404 shutdown_connections 2024-05-09T05:00:13.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.841+0000 7f30967e4640 1 -- 172.21.15.60:0/1298011404 wait complete. 2024-05-09T05:00:13.855 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-05-09T05:00:13.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.972+0000 7f4402bc8640 1 Processor -- start 2024-05-09T05:00:13.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.972+0000 7f4402bc8640 1 -- start start 2024-05-09T05:00:13.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.973+0000 7f4402bc8640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f43fc0b9e50 0x7f43fc0ba300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.973+0000 7f4402bc8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f43fc0b54b0 0x7f43fc0b5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.973+0000 7f4402bc8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43fc0b7be0 con 0x7f43fc148dc0 2024-05-09T05:00:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.973+0000 7f4402bc8640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f43fc160300 con 0x7f43fc0b54b0 2024-05-09T05:00:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.973+0000 7f4402bc8640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f43fc15fcb0 con 0x7f43fc0b9e50 2024-05-09T05:00:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.973+0000 7f4401165640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f43fc0b54b0 0x7f43fc0b5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.973+0000 7f4401165640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f43fc0b54b0 0x7f43fc0b5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40502/0 (socket says 172.21.15.60:40502) 2024-05-09T05:00:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.973+0000 7f4401165640 1 -- 172.21.15.60:0/1188939125 learned_addr learned my addr 172.21.15.60:0/1188939125 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.974+0000 7f43fbfff640 1 --2- 172.21.15.60:0/1188939125 >> v2:172.21.15.114:3300/0 conn(0x7f43fc0b9e50 0x7f43fc0ba300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.974+0000 7f4401165640 1 -- 172.21.15.60:0/1188939125 >> v2:172.21.15.114:3300/0 conn(0x7f43fc0b9e50 msgr2=0x7f43fc0ba300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.974+0000 7f4401165640 1 --2- 172.21.15.60:0/1188939125 >> v2:172.21.15.114:3300/0 conn(0x7f43fc0b9e50 0x7f43fc0ba300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.974+0000 7f4401165640 1 -- 172.21.15.60:0/1188939125 >> v1:172.21.15.114:6789/0 conn(0x7f43fc148dc0 legacy=0x7f43fc0b7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.974+0000 7f4401165640 1 -- 172.21.15.60:0/1188939125 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43fc15ff90 con 0x7f43fc0b54b0 2024-05-09T05:00:13.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.974+0000 7f43fbfff640 1 --2- 172.21.15.60:0/1188939125 >> v2:172.21.15.114:3300/0 conn(0x7f43fc0b9e50 0x7f43fc0ba300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:13.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.974+0000 7f4401165640 1 --2- 172.21.15.60:0/1188939125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43fc0b54b0 0x7f43fc0b5940 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f43fc0d2160 tx=0x7f43ec02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5660570b44656b06 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:13.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.975+0000 7f43fb7fe640 1 -- 172.21.15.60:0/1188939125 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec037070 con 0x7f43fc0b54b0 2024-05-09T05:00:13.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.975+0000 7f43fb7fe640 1 -- 172.21.15.60:0/1188939125 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43ec032440 con 0x7f43fc0b54b0 2024-05-09T05:00:13.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.975+0000 7f43fb7fe640 1 -- 172.21.15.60:0/1188939125 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec032810 con 0x7f43fc0b54b0 2024-05-09T05:00:13.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.975+0000 7f4402bc8640 1 -- 172.21.15.60:0/1188939125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43fc0b54b0 msgr2=0x7f43fc0b5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:13.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.975+0000 7f4402bc8640 1 --2- 172.21.15.60:0/1188939125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43fc0b54b0 0x7f43fc0b5940 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f43fc0d2160 tx=0x7f43ec02a1e0 comp rx=0 tx=0).stop 2024-05-09T05:00:13.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.975+0000 7f4402bc8640 1 -- 172.21.15.60:0/1188939125 shutdown_connections 2024-05-09T05:00:13.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.975+0000 7f4402bc8640 1 --2- 172.21.15.60:0/1188939125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43fc0b54b0 0x7f43fc0b5940 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:13.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.975+0000 7f4402bc8640 1 --2- 172.21.15.60:0/1188939125 >> v2:172.21.15.114:3300/0 conn(0x7f43fc0b9e50 0x7f43fc0ba300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:13.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.975+0000 7f4402bc8640 1 -- 172.21.15.60:0/1188939125 >> 172.21.15.60:0/1188939125 conn(0x7f43fc06cf70 msgr2=0x7f43fc0b85e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:13.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.975+0000 7f4402bc8640 1 -- 172.21.15.60:0/1188939125 shutdown_connections 2024-05-09T05:00:13.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.975+0000 7f4402bc8640 1 -- 172.21.15.60:0/1188939125 wait complete. 2024-05-09T05:00:13.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.976+0000 7f4402bc8640 1 Processor -- start 2024-05-09T05:00:13.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.976+0000 7f4402bc8640 1 -- start start 2024-05-09T05:00:13.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.977+0000 7f4402bc8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43fc0b54b0 0x7f43fc1f7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:13.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.977+0000 7f4402bc8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43fc0b9e50 0x7f43fc0af2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:13.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.977+0000 7f4402bc8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43fc148dc0 0x7f43fc1f9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:13.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.977+0000 7f4402bc8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f43fc15f960 con 0x7f43fc148dc0 2024-05-09T05:00:13.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.977+0000 7f4402bc8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f43fc1491a0 con 0x7f43fc0b9e50 2024-05-09T05:00:13.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.977+0000 7f4402bc8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f43fc160790 con 0x7f43fc0b54b0 2024-05-09T05:00:13.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.977+0000 7f4401165640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43fc148dc0 0x7f43fc1f9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:13.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.977+0000 7f4401165640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43fc148dc0 0x7f43fc1f9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40510/0 (socket says 172.21.15.60:40510) 2024-05-09T05:00:13.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.977+0000 7f4401165640 1 -- 172.21.15.60:0/3946815696 learned_addr learned my addr 172.21.15.60:0/3946815696 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:13.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.978+0000 7f43fbfff640 1 --2- 172.21.15.60:0/3946815696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43fc0b9e50 0x7f43fc0af2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:13.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.978+0000 7f4400964640 1 --2- 172.21.15.60:0/3946815696 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43fc0b54b0 0x7f43fc1f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:13.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.978+0000 7f4401165640 1 -- 172.21.15.60:0/3946815696 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43fc0b54b0 msgr2=0x7f43fc1f7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:13.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.978+0000 7f4401165640 1 --2- 172.21.15.60:0/3946815696 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43fc0b54b0 0x7f43fc1f7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:13.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.978+0000 7f4401165640 1 -- 172.21.15.60:0/3946815696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43fc0b9e50 msgr2=0x7f43fc0af2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:13.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.978+0000 7f4401165640 1 --2- 172.21.15.60:0/3946815696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43fc0b9e50 0x7f43fc0af2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:13.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.978+0000 7f4401165640 1 -- 172.21.15.60:0/3946815696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43fc161ff0 con 0x7f43fc148dc0 2024-05-09T05:00:13.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.978+0000 7f4401165640 1 --2- 172.21.15.60:0/3946815696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43fc148dc0 0x7f43fc1f9170 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f43ec0090b0 tx=0x7f43ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:13.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.978+0000 7f43f97fa640 1 -- 172.21.15.60:0/3946815696 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec041020 con 0x7f43fc148dc0 2024-05-09T05:00:13.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.978+0000 7f43f97fa640 1 -- 172.21.15.60:0/3946815696 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43ec030990 con 0x7f43fc148dc0 2024-05-09T05:00:13.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.978+0000 7f4402bc8640 1 -- 172.21.15.60:0/3946815696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43fc162260 con 0x7f43fc148dc0 2024-05-09T05:00:13.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.978+0000 7f43f97fa640 1 -- 172.21.15.60:0/3946815696 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec037040 con 0x7f43fc148dc0 2024-05-09T05:00:13.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.978+0000 7f4402bc8640 1 -- 172.21.15.60:0/3946815696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43fc162820 con 0x7f43fc148dc0 2024-05-09T05:00:13.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.979+0000 7f4402bc8640 1 -- 172.21.15.60:0/3946815696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43fc0d24f0 con 0x7f43fc148dc0 2024-05-09T05:00:13.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.980+0000 7f43f97fa640 1 -- 172.21.15.60:0/3946815696 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f43ec044020 con 0x7f43fc148dc0 2024-05-09T05:00:13.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.980+0000 7f43f97fa640 1 --2- 172.21.15.60:0/3946815696 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f43e8076300 0x7f43e80787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:13.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.980+0000 7f43f97fa640 1 -- 172.21.15.60:0/3946815696 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 6007+0+0 (secure 0 0 0) 0x7f43ec0b8010 con 0x7f43fc148dc0 2024-05-09T05:00:13.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.981+0000 7f4400964640 1 --2- 172.21.15.60:0/3946815696 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f43e8076300 0x7f43e80787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:13.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.981+0000 7f4400964640 1 --2- 172.21.15.60:0/3946815696 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f43e8076300 0x7f43e80787c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f43f4005520 tx=0x7f43f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:13.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:13.985+0000 7f43f97fa640 1 -- 172.21.15.60:0/3946815696 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f43ec04b030 con 0x7f43fc148dc0 2024-05-09T05:00:14.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:14.174+0000 7f4402bc8640 1 -- 172.21.15.60:0/3946815696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f43fc075b30 con 0x7f43fc148dc0 2024-05-09T05:00:15.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.848+0000 7f43f97fa640 1 -- 172.21.15.60:0/3946815696 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) ==== 228+0+0 (secure 0 0 0) 0x7f43fc075b30 con 0x7f43fc148dc0 2024-05-09T05:00:15.849 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data_ec' created 2024-05-09T05:00:15.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.853+0000 7f4402bc8640 1 -- 172.21.15.60:0/3946815696 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f43e8076300 msgr2=0x7f43e80787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:15.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.853+0000 7f4402bc8640 1 --2- 172.21.15.60:0/3946815696 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f43e8076300 0x7f43e80787c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f43f4005520 tx=0x7f43f4009040 comp rx=0 tx=0).stop 2024-05-09T05:00:15.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.853+0000 7f4402bc8640 1 -- 172.21.15.60:0/3946815696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43fc148dc0 msgr2=0x7f43fc1f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:15.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.853+0000 7f4402bc8640 1 --2- 172.21.15.60:0/3946815696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43fc148dc0 0x7f43fc1f9170 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f43ec0090b0 tx=0x7f43ec0046c0 comp rx=0 tx=0).stop 2024-05-09T05:00:15.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.853+0000 7f4402bc8640 1 -- 172.21.15.60:0/3946815696 shutdown_connections 2024-05-09T05:00:15.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.853+0000 7f4402bc8640 1 --2- 172.21.15.60:0/3946815696 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f43e8076300 0x7f43e80787c0 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f43f4005520 tx=0x7f43f4009040 comp rx=0 tx=0).stop 2024-05-09T05:00:15.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.853+0000 7f4402bc8640 1 --2- 172.21.15.60:0/3946815696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43fc148dc0 0x7f43fc1f9170 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:15.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.853+0000 7f4402bc8640 1 --2- 172.21.15.60:0/3946815696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43fc0b9e50 0x7f43fc0af2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:15.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.853+0000 7f4402bc8640 1 --2- 172.21.15.60:0/3946815696 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f43fc0b54b0 0x7f43fc1f7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:15.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.853+0000 7f4402bc8640 1 -- 172.21.15.60:0/3946815696 >> 172.21.15.60:0/3946815696 conn(0x7f43fc06cf70 msgr2=0x7f43fc0b8df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:15.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.853+0000 7f4402bc8640 1 -- 172.21.15.60:0/3946815696 shutdown_connections 2024-05-09T05:00:15.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.853+0000 7f4402bc8640 1 -- 172.21.15.60:0/3946815696 wait complete. 2024-05-09T05:00:15.873 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-05-09T05:00:15.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.986+0000 7f6d55220640 1 Processor -- start 2024-05-09T05:00:15.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.986+0000 7f6d55220640 1 -- start start 2024-05-09T05:00:15.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.987+0000 7f6d55220640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d5015f0f0 0x7f6d500b8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:15.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.987+0000 7f6d55220640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6d500d2af0 0x7f6d500d2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:15.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.987+0000 7f6d55220640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d500b5550 con 0x7f6d5015f4d0 2024-05-09T05:00:15.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.987+0000 7f6d55220640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f6d500d3490 con 0x7f6d500d2af0 2024-05-09T05:00:15.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.987+0000 7f6d55220640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6d500b6160 con 0x7f6d5015f0f0 2024-05-09T05:00:15.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.987+0000 7f6d4e59c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6d5015f4d0 0x7f6d500b65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43544/0 (socket says 172.21.15.60:43544) 2024-05-09T05:00:15.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.987+0000 7f6d4e59c640 1 -- 172.21.15.60:0/71823441 learned_addr learned my addr 172.21.15.60:0/71823441 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:15.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.987+0000 7f6d4ed9d640 1 --2- 172.21.15.60:0/71823441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d5015f0f0 0x7f6d500b8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:15.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.987+0000 7f6d4f59e640 1 --2- 172.21.15.60:0/71823441 >> v2:172.21.15.114:3300/0 conn(0x7f6d500d2af0 0x7f6d500d2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:15.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.987+0000 7f6d4dd9b640 1 -- 172.21.15.60:0/71823441 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3246366724 0 0) 0x7f6d500b5550 con 0x7f6d5015f4d0 2024-05-09T05:00:15.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.988+0000 7f6d4dd9b640 1 -- 172.21.15.60:0/71823441 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d3c003520 con 0x7f6d5015f4d0 2024-05-09T05:00:15.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.988+0000 7f6d4ed9d640 1 -- 172.21.15.60:0/71823441 >> v2:172.21.15.114:3300/0 conn(0x7f6d500d2af0 msgr2=0x7f6d500d2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:15.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.988+0000 7f6d4ed9d640 1 --2- 172.21.15.60:0/71823441 >> v2:172.21.15.114:3300/0 conn(0x7f6d500d2af0 0x7f6d500d2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:15.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.988+0000 7f6d4ed9d640 1 -- 172.21.15.60:0/71823441 >> v1:172.21.15.60:6789/0 conn(0x7f6d5015f4d0 legacy=0x7f6d500b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:15.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.988+0000 7f6d4ed9d640 1 -- 172.21.15.60:0/71823441 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d500b5830 con 0x7f6d5015f0f0 2024-05-09T05:00:15.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.988+0000 7f6d4f59e640 1 --2- 172.21.15.60:0/71823441 >> v2:172.21.15.114:3300/0 conn(0x7f6d500d2af0 0x7f6d500d2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:00:15.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.988+0000 7f6d4ed9d640 1 --2- 172.21.15.60:0/71823441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d5015f0f0 0x7f6d500b8ef0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6d480097a0 tx=0x7f6d4802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f65687b8fc7ab54b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:15.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.989+0000 7f6d4dd9b640 1 -- 172.21.15.60:0/71823441 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d48037070 con 0x7f6d5015f0f0 2024-05-09T05:00:15.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.989+0000 7f6d4dd9b640 1 -- 172.21.15.60:0/71823441 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d48032440 con 0x7f6d5015f0f0 2024-05-09T05:00:15.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.989+0000 7f6d4dd9b640 1 -- 172.21.15.60:0/71823441 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d480327f0 con 0x7f6d5015f0f0 2024-05-09T05:00:15.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.989+0000 7f6d55220640 1 -- 172.21.15.60:0/71823441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d5015f0f0 msgr2=0x7f6d500b8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:15.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.989+0000 7f6d55220640 1 --2- 172.21.15.60:0/71823441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d5015f0f0 0x7f6d500b8ef0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6d480097a0 tx=0x7f6d4802a180 comp rx=0 tx=0).stop 2024-05-09T05:00:15.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.989+0000 7f6d55220640 1 -- 172.21.15.60:0/71823441 shutdown_connections 2024-05-09T05:00:15.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.989+0000 7f6d55220640 1 --2- 172.21.15.60:0/71823441 >> v2:172.21.15.114:3300/0 conn(0x7f6d500d2af0 0x7f6d500d2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:15.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.989+0000 7f6d55220640 1 --2- 172.21.15.60:0/71823441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d5015f0f0 0x7f6d500b8ef0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:15.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.989+0000 7f6d55220640 1 -- 172.21.15.60:0/71823441 >> 172.21.15.60:0/71823441 conn(0x7f6d5006ebd0 msgr2=0x7f6d50073840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:15.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.989+0000 7f6d55220640 1 -- 172.21.15.60:0/71823441 shutdown_connections 2024-05-09T05:00:15.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.989+0000 7f6d55220640 1 -- 172.21.15.60:0/71823441 wait complete. 2024-05-09T05:00:15.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.991+0000 7f6d55220640 1 Processor -- start 2024-05-09T05:00:15.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.991+0000 7f6d55220640 1 -- start start 2024-05-09T05:00:15.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.991+0000 7f6d55220640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d500d2af0 0x7f6d501579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:15.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.991+0000 7f6d55220640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d5015f0f0 0x7f6d50073340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:15.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.991+0000 7f6d55220640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d5015f4d0 0x7f6d50150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:15.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.991+0000 7f6d55220640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6d500b9430 con 0x7f6d500d2af0 2024-05-09T05:00:15.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.991+0000 7f6d55220640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6d500b5df0 con 0x7f6d5015f0f0 2024-05-09T05:00:15.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.991+0000 7f6d55220640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6d500b95b0 con 0x7f6d5015f4d0 2024-05-09T05:00:15.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.992+0000 7f6d4ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d500d2af0 0x7f6d501579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:15.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.992+0000 7f6d4ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d500d2af0 0x7f6d501579c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40524/0 (socket says 172.21.15.60:40524) 2024-05-09T05:00:15.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.992+0000 7f6d4ed9d640 1 -- 172.21.15.60:0/1502135720 learned_addr learned my addr 172.21.15.60:0/1502135720 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:15.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.992+0000 7f6d4e59c640 1 --2- 172.21.15.60:0/1502135720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d5015f0f0 0x7f6d50073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:15.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.992+0000 7f6d4f59e640 1 --2- 172.21.15.60:0/1502135720 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d5015f4d0 0x7f6d50150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:15.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.992+0000 7f6d4ed9d640 1 -- 172.21.15.60:0/1502135720 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d5015f4d0 msgr2=0x7f6d50150930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:15.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.992+0000 7f6d4ed9d640 1 --2- 172.21.15.60:0/1502135720 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d5015f4d0 0x7f6d50150930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:15.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.992+0000 7f6d4ed9d640 1 -- 172.21.15.60:0/1502135720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d5015f0f0 msgr2=0x7f6d50073340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:15.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.992+0000 7f6d4ed9d640 1 --2- 172.21.15.60:0/1502135720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d5015f0f0 0x7f6d50073340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:15.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.992+0000 7f6d4ed9d640 1 -- 172.21.15.60:0/1502135720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d50157f00 con 0x7f6d500d2af0 2024-05-09T05:00:15.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.992+0000 7f6d4ed9d640 1 --2- 172.21.15.60:0/1502135720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d500d2af0 0x7f6d501579c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6d48035040 tx=0x7f6d480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:15.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.992+0000 7f6d477fe640 1 -- 172.21.15.60:0/1502135720 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d48037050 con 0x7f6d500d2af0 2024-05-09T05:00:15.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.992+0000 7f6d477fe640 1 -- 172.21.15.60:0/1502135720 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d48043070 con 0x7f6d500d2af0 2024-05-09T05:00:15.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.992+0000 7f6d55220640 1 -- 172.21.15.60:0/1502135720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d5015f960 con 0x7f6d500d2af0 2024-05-09T05:00:15.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.992+0000 7f6d477fe640 1 -- 172.21.15.60:0/1502135720 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d48006e10 con 0x7f6d500d2af0 2024-05-09T05:00:15.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.992+0000 7f6d55220640 1 -- 172.21.15.60:0/1502135720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d501601a0 con 0x7f6d500d2af0 2024-05-09T05:00:15.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.993+0000 7f6d55220640 1 -- 172.21.15.60:0/1502135720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d50051040 con 0x7f6d500d2af0 2024-05-09T05:00:15.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.994+0000 7f6d477fe640 1 -- 172.21.15.60:0/1502135720 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6d48032810 con 0x7f6d500d2af0 2024-05-09T05:00:15.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.994+0000 7f6d477fe640 1 --2- 172.21.15.60:0/1502135720 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6d300762d0 0x7f6d30078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:15.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.995+0000 7f6d477fe640 1 -- 172.21.15.60:0/1502135720 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 6500+0+0 (secure 0 0 0) 0x7f6d480b8280 con 0x7f6d500d2af0 2024-05-09T05:00:15.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.995+0000 7f6d4e59c640 1 --2- 172.21.15.60:0/1502135720 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6d300762d0 0x7f6d30078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:15.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.995+0000 7f6d4e59c640 1 --2- 172.21.15.60:0/1502135720 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6d300762d0 0x7f6d30078790 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6d50151b80 tx=0x7f6d38006aa0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:15.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:15.999+0000 7f6d477fe640 1 -- 172.21.15.60:0/1502135720 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6d5015f960 con 0x7f6d500d2af0 2024-05-09T05:00:16.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.189+0000 7f6d55220640 1 -- 172.21.15.60:0/1502135720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f6d5006f850 con 0x7f6d500d2af0 2024-05-09T05:00:16.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.850+0000 7f6d477fe640 1 -- 172.21.15.60:0/1502135720 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) ==== 169+0+0 (secure 0 0 0) 0x7f6d480817a0 con 0x7f6d500d2af0 2024-05-09T05:00:16.851 INFO:teuthology.orchestra.run.smithi060.stderr:set pool 4 allow_ec_overwrites to true 2024-05-09T05:00:16.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.853+0000 7f6d55220640 1 -- 172.21.15.60:0/1502135720 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6d300762d0 msgr2=0x7f6d30078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:16.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.853+0000 7f6d55220640 1 --2- 172.21.15.60:0/1502135720 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6d300762d0 0x7f6d30078790 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6d50151b80 tx=0x7f6d38006aa0 comp rx=0 tx=0).stop 2024-05-09T05:00:16.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.853+0000 7f6d55220640 1 -- 172.21.15.60:0/1502135720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d500d2af0 msgr2=0x7f6d501579c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:16.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.853+0000 7f6d55220640 1 --2- 172.21.15.60:0/1502135720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d500d2af0 0x7f6d501579c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6d48035040 tx=0x7f6d480046c0 comp rx=0 tx=0).stop 2024-05-09T05:00:16.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.853+0000 7f6d55220640 1 -- 172.21.15.60:0/1502135720 shutdown_connections 2024-05-09T05:00:16.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.854+0000 7f6d55220640 1 --2- 172.21.15.60:0/1502135720 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6d300762d0 0x7f6d30078790 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:16.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.854+0000 7f6d55220640 1 --2- 172.21.15.60:0/1502135720 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d5015f4d0 0x7f6d50150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:16.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.854+0000 7f6d55220640 1 --2- 172.21.15.60:0/1502135720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d5015f0f0 0x7f6d50073340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:16.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.854+0000 7f6d55220640 1 --2- 172.21.15.60:0/1502135720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d500d2af0 0x7f6d501579c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:16.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.854+0000 7f6d55220640 1 -- 172.21.15.60:0/1502135720 >> 172.21.15.60:0/1502135720 conn(0x7f6d5006ebd0 msgr2=0x7f6d500b9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:16.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.854+0000 7f6d55220640 1 -- 172.21.15.60:0/1502135720 shutdown_connections 2024-05-09T05:00:16.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.854+0000 7f6d55220640 1 -- 172.21.15.60:0/1502135720 wait complete. 2024-05-09T05:00:16.886 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-05-09T05:00:16.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.997+0000 7f132e4f1640 1 Processor -- start 2024-05-09T05:00:16.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.997+0000 7f132e4f1640 1 -- start start 2024-05-09T05:00:16.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.998+0000 7f132e4f1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132815f490 0x7f132806fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:17.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.998+0000 7f132e4f1640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1328070080 0x7f13280d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:17.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.998+0000 7f132e4f1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f132806f6a0 con 0x7f13280d2980 2024-05-09T05:00:17.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.998+0000 7f132e4f1640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f13280704e0 con 0x7f1328070080 2024-05-09T05:00:17.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.998+0000 7f132e4f1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f132806e8e0 con 0x7f132815f490 2024-05-09T05:00:17.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.999+0000 7f1327fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132815f490 0x7f132806fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:17.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.999+0000 7f13277fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1328070080 0x7f13280d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:17.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.999+0000 7f132ca8e640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f13280d2980 0x7f13281630e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.60:51052/0 (socket says 172.21.15.60:51052) 2024-05-09T05:00:17.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.999+0000 7f132ca8e640 1 -- 172.21.15.60:0/3872177730 learned_addr learned my addr 172.21.15.60:0/3872177730 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:17.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.999+0000 7f13277fe640 1 -- 172.21.15.60:0/3872177730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132815f490 msgr2=0x7f132806fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:17.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.999+0000 7f13277fe640 1 --2- 172.21.15.60:0/3872177730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132815f490 0x7f132806fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:17.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.999+0000 7f13277fe640 1 -- 172.21.15.60:0/3872177730 >> v1:172.21.15.114:6789/0 conn(0x7f13280d2980 legacy=0x7f13281630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:17.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:16.999+0000 7f13277fe640 1 -- 172.21.15.60:0/3872177730 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132806ebc0 con 0x7f1328070080 2024-05-09T05:00:17.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.000+0000 7f13277fe640 1 --2- 172.21.15.60:0/3872177730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1328070080 0x7f13280d2440 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f131c0097a0 tx=0x7f131c02a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ded28289a1a9cef1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:17.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.000+0000 7f1326ffd640 1 -- 172.21.15.60:0/3872177730 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c037070 con 0x7f1328070080 2024-05-09T05:00:17.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.000+0000 7f1326ffd640 1 -- 172.21.15.60:0/3872177730 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f131c02ae50 con 0x7f1328070080 2024-05-09T05:00:17.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.000+0000 7f1326ffd640 1 -- 172.21.15.60:0/3872177730 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c032660 con 0x7f1328070080 2024-05-09T05:00:17.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.000+0000 7f132e4f1640 1 -- 172.21.15.60:0/3872177730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1328070080 msgr2=0x7f13280d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:17.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.000+0000 7f132e4f1640 1 --2- 172.21.15.60:0/3872177730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1328070080 0x7f13280d2440 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f131c0097a0 tx=0x7f131c02a100 comp rx=0 tx=0).stop 2024-05-09T05:00:17.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.000+0000 7f132e4f1640 1 -- 172.21.15.60:0/3872177730 shutdown_connections 2024-05-09T05:00:17.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.000+0000 7f132e4f1640 1 --2- 172.21.15.60:0/3872177730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1328070080 0x7f13280d2440 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:17.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.000+0000 7f132e4f1640 1 --2- 172.21.15.60:0/3872177730 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132815f490 0x7f132806fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:17.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.000+0000 7f132e4f1640 1 -- 172.21.15.60:0/3872177730 >> 172.21.15.60:0/3872177730 conn(0x7f132809b8c0 msgr2=0x7f132809bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:17.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.000+0000 7f132e4f1640 1 -- 172.21.15.60:0/3872177730 shutdown_connections 2024-05-09T05:00:17.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.000+0000 7f132e4f1640 1 -- 172.21.15.60:0/3872177730 wait complete. 2024-05-09T05:00:17.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.001+0000 7f132e4f1640 1 Processor -- start 2024-05-09T05:00:17.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.002+0000 7f132e4f1640 1 -- start start 2024-05-09T05:00:17.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.002+0000 7f132e4f1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1328070080 0x7f13281ea9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:17.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.002+0000 7f132e4f1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13280d2980 0x7f13281eaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:17.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.002+0000 7f132e4f1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132815f490 0x7f13281e8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:17.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.002+0000 7f132e4f1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f132815f870 con 0x7f1328070080 2024-05-09T05:00:17.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.002+0000 7f132e4f1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f13280d2e10 con 0x7f13280d2980 2024-05-09T05:00:17.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.002+0000 7f132e4f1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1328073860 con 0x7f132815f490 2024-05-09T05:00:17.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.003+0000 7f1327fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1328070080 0x7f13281ea9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:17.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.003+0000 7f13277fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13280d2980 0x7f13281eaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:17.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.003+0000 7f13277fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13280d2980 0x7f13281eaee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.60:55804/0 (socket says 172.21.15.60:55804) 2024-05-09T05:00:17.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.003+0000 7f13277fe640 1 -- 172.21.15.60:0/1262328772 learned_addr learned my addr 172.21.15.60:0/1262328772 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:17.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.003+0000 7f132ca8e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132815f490 0x7f13281e8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:17.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.003+0000 7f13277fe640 1 -- 172.21.15.60:0/1262328772 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132815f490 msgr2=0x7f13281e8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:17.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.003+0000 7f13277fe640 1 --2- 172.21.15.60:0/1262328772 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132815f490 0x7f13281e8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:17.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.003+0000 7f13277fe640 1 -- 172.21.15.60:0/1262328772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1328070080 msgr2=0x7f13281ea9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:17.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.003+0000 7f13277fe640 1 --2- 172.21.15.60:0/1262328772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1328070080 0x7f13281ea9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:17.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.003+0000 7f13277fe640 1 -- 172.21.15.60:0/1262328772 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13280ae370 con 0x7f13280d2980 2024-05-09T05:00:17.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.003+0000 7f13277fe640 1 --2- 172.21.15.60:0/1262328772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13280d2980 0x7f13281eaee0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f131c0098c0 tx=0x7f131c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:17.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.003+0000 7f1324ff9640 1 -- 172.21.15.60:0/1262328772 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c041030 con 0x7f13280d2980 2024-05-09T05:00:17.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.003+0000 7f1324ff9640 1 -- 172.21.15.60:0/1262328772 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f131c02ac50 con 0x7f13280d2980 2024-05-09T05:00:17.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.003+0000 7f132e4f1640 1 -- 172.21.15.60:0/1262328772 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13280ae650 con 0x7f13280d2980 2024-05-09T05:00:17.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.003+0000 7f1324ff9640 1 -- 172.21.15.60:0/1262328772 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c037040 con 0x7f13280d2980 2024-05-09T05:00:17.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.003+0000 7f132e4f1640 1 -- 172.21.15.60:0/1262328772 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13280aec10 con 0x7f13280d2980 2024-05-09T05:00:17.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.004+0000 7f132e4f1640 1 -- 172.21.15.60:0/1262328772 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1328051040 con 0x7f13280d2980 2024-05-09T05:00:17.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.005+0000 7f1324ff9640 1 -- 172.21.15.60:0/1262328772 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f131c044070 con 0x7f13280d2980 2024-05-09T05:00:17.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.005+0000 7f1324ff9640 1 --2- 172.21.15.60:0/1262328772 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f13140762d0 0x7f1314078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:17.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.006+0000 7f1324ff9640 1 -- 172.21.15.60:0/1262328772 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7f131c0b7de0 con 0x7f13280d2980 2024-05-09T05:00:17.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.006+0000 7f1327fff640 1 --2- 172.21.15.60:0/1262328772 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f13140762d0 0x7f1314078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:17.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.006+0000 7f1327fff640 1 --2- 172.21.15.60:0/1262328772 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f13140762d0 0x7f1314078790 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1318009b40 tx=0x7f1318009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:17.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.010+0000 7f1324ff9640 1 -- 172.21.15.60:0/1262328772 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f131c080dd0 con 0x7f13280d2980 2024-05-09T05:00:17.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.246+0000 7f132e4f1640 1 -- 172.21.15.60:0/1262328772 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f1328073640 con 0x7f13280d2980 2024-05-09T05:00:17.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.863+0000 7f1324ff9640 1 -- 172.21.15.60:0/1262328772 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) ==== 137+0+0 (secure 0 0 0) 0x7f131c02ae20 con 0x7f13280d2980 2024-05-09T05:00:17.865 INFO:teuthology.orchestra.run.smithi060.stderr:added data pool 4 to fsmap 2024-05-09T05:00:17.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.866+0000 7f132e4f1640 1 -- 172.21.15.60:0/1262328772 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f13140762d0 msgr2=0x7f1314078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:17.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.866+0000 7f132e4f1640 1 --2- 172.21.15.60:0/1262328772 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f13140762d0 0x7f1314078790 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1318009b40 tx=0x7f1318009190 comp rx=0 tx=0).stop 2024-05-09T05:00:17.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.866+0000 7f132e4f1640 1 -- 172.21.15.60:0/1262328772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13280d2980 msgr2=0x7f13281eaee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:17.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.866+0000 7f132e4f1640 1 --2- 172.21.15.60:0/1262328772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13280d2980 0x7f13281eaee0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f131c0098c0 tx=0x7f131c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:00:17.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.866+0000 7f132e4f1640 1 -- 172.21.15.60:0/1262328772 shutdown_connections 2024-05-09T05:00:17.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.866+0000 7f132e4f1640 1 --2- 172.21.15.60:0/1262328772 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f13140762d0 0x7f1314078790 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:17.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.866+0000 7f132e4f1640 1 --2- 172.21.15.60:0/1262328772 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f132815f490 0x7f13281e8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:17.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.866+0000 7f132e4f1640 1 --2- 172.21.15.60:0/1262328772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13280d2980 0x7f13281eaee0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:17.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.866+0000 7f132e4f1640 1 --2- 172.21.15.60:0/1262328772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1328070080 0x7f13281ea9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:17.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.866+0000 7f132e4f1640 1 -- 172.21.15.60:0/1262328772 >> 172.21.15.60:0/1262328772 conn(0x7f132809b8c0 msgr2=0x7f13281ec150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:17.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.866+0000 7f132e4f1640 1 -- 172.21.15.60:0/1262328772 shutdown_connections 2024-05-09T05:00:17.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.866+0000 7f132e4f1640 1 -- 172.21.15.60:0/1262328772 wait complete. 2024-05-09T05:00:17.881 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:00:17.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.990+0000 7fb29fb7f640 1 Processor -- start 2024-05-09T05:00:17.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.990+0000 7fb29fb7f640 1 -- start start 2024-05-09T05:00:17.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.991+0000 7fb29fb7f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2981606f0 0x7fb2980d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:17.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.991+0000 7fb29fb7f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb29806ff60 con 0x7fb29815f300 2024-05-09T05:00:17.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.991+0000 7fb29fb7f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb298070240 con 0x7fb29806f9f0 2024-05-09T05:00:17.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.991+0000 7fb29fb7f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb298070520 con 0x7fb2981606f0 2024-05-09T05:00:17.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.991+0000 7fb29d91b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb29815f300 0x7fb2981639f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43558/0 (socket says 172.21.15.60:43558) 2024-05-09T05:00:17.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.991+0000 7fb29d91b640 1 -- 172.21.15.60:0/1700677587 learned_addr learned my addr 172.21.15.60:0/1700677587 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:17.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.991+0000 7fb29c919640 1 -- 172.21.15.60:0/1700677587 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1394548645 0 0) 0x7fb29806ff60 con 0x7fb29815f300 2024-05-09T05:00:17.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.991+0000 7fb29c919640 1 -- 172.21.15.60:0/1700677587 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb284003520 con 0x7fb29815f300 2024-05-09T05:00:17.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.991+0000 7fb29d11a640 1 --2- 172.21.15.60:0/1700677587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2981606f0 0x7fb2980d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:17.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29c919640 1 -- 172.21.15.60:0/1700677587 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 546661740 0 0) 0x7fb284003520 con 0x7fb29815f300 2024-05-09T05:00:17.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29c919640 1 -- 172.21.15.60:0/1700677587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2981606f0 msgr2=0x7fb2980d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:17.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29c919640 1 --2- 172.21.15.60:0/1700677587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2981606f0 0x7fb2980d33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:17.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29c919640 1 -- 172.21.15.60:0/1700677587 >> v1:172.21.15.114:6789/0 conn(0x7fb29806f9f0 legacy=0x7fb29806ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:17.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29c919640 1 -- 172.21.15.60:0/1700677587 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb29806e740 con 0x7fb29815f300 2024-05-09T05:00:17.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29c919640 1 -- 172.21.15.60:0/1700677587 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7fb2940016f0 con 0x7fb29815f300 2024-05-09T05:00:17.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29d11a640 1 --2- 172.21.15.60:0/1700677587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2981606f0 0x7fb2980d33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:17.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29c919640 1 -- 172.21.15.60:0/1700677587 >> v1:172.21.15.60:6789/0 conn(0x7fb29815f300 legacy=0x7fb2981639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:17.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29c919640 1 --2- 172.21.15.60:0/1700677587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2981606f0 0x7fb284003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:17.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29c919640 1 --2- 172.21.15.60:0/1700677587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb284004070 0x7fb284004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:17.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29c919640 1 --2- 172.21.15.60:0/1700677587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb284004d40 0x7fb2840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:17.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29c919640 1 -- 172.21.15.60:0/1700677587 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb298070520 con 0x7fb284004d40 2024-05-09T05:00:17.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29c919640 1 -- 172.21.15.60:0/1700677587 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb28400bbb0 con 0x7fb284004070 2024-05-09T05:00:17.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29c919640 1 -- 172.21.15.60:0/1700677587 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb28400bdb0 con 0x7fb2981606f0 2024-05-09T05:00:17.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29e11c640 1 --2- 172.21.15.60:0/1700677587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb284004d40 0x7fb2840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:17.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29d91b640 1 --2- 172.21.15.60:0/1700677587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2981606f0 0x7fb284003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:17.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29d11a640 1 --2- 172.21.15.60:0/1700677587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb284004070 0x7fb284004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:17.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29e11c640 1 -- 172.21.15.60:0/1700677587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2981606f0 msgr2=0x7fb284003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:17.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29e11c640 1 --2- 172.21.15.60:0/1700677587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb2981606f0 0x7fb284003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:17.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29e11c640 1 -- 172.21.15.60:0/1700677587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb284004070 msgr2=0x7fb284004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:17.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29e11c640 1 --2- 172.21.15.60:0/1700677587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb284004070 0x7fb284004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:17.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.992+0000 7fb29e11c640 1 -- 172.21.15.60:0/1700677587 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb284003520 con 0x7fb284004d40 2024-05-09T05:00:17.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.993+0000 7fb29e11c640 1 --2- 172.21.15.60:0/1700677587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb284004d40 0x7fb2840051c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb290001e90 tx=0x7fb284005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=305221ad7efa3984 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:17.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.993+0000 7fb29c919640 1 -- 172.21.15.60:0/1700677587 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb290001ab0 con 0x7fb284004d40 2024-05-09T05:00:17.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.993+0000 7fb29c919640 1 -- 172.21.15.60:0/1700677587 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb29002ae50 con 0x7fb284004d40 2024-05-09T05:00:17.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.993+0000 7fb29c919640 1 -- 172.21.15.60:0/1700677587 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb290003b70 con 0x7fb284004d40 2024-05-09T05:00:17.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.993+0000 7fb29fb7f640 1 -- 172.21.15.60:0/1700677587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb284004d40 msgr2=0x7fb2840051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:17.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.994+0000 7fb29fb7f640 1 --2- 172.21.15.60:0/1700677587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb284004d40 0x7fb2840051c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb290001e90 tx=0x7fb284005830 comp rx=0 tx=0).stop 2024-05-09T05:00:17.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.994+0000 7fb29e11c640 1 -- 172.21.15.60:0/1700677587 reap_dead start 2024-05-09T05:00:17.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.994+0000 7fb29fb7f640 1 -- 172.21.15.60:0/1700677587 shutdown_connections 2024-05-09T05:00:17.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.994+0000 7fb29fb7f640 1 -- 172.21.15.60:0/1700677587 >> 172.21.15.60:0/1700677587 conn(0x7fb29809b850 msgr2=0x7fb29809bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:17.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.994+0000 7fb29fb7f640 1 -- 172.21.15.60:0/1700677587 shutdown_connections 2024-05-09T05:00:17.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.994+0000 7fb29fb7f640 1 -- 172.21.15.60:0/1700677587 wait complete. 2024-05-09T05:00:18.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.995+0000 7fb29fb7f640 1 Processor -- start 2024-05-09T05:00:18.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.995+0000 7fb29fb7f640 1 -- start start 2024-05-09T05:00:18.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb29fb7f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb284004d40 0x7fb2980ac760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:18.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb29fb7f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb29809ef20 0x7fb2981f8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:18.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb29fb7f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb298162b80 0x7fb298163000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:18.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb29fb7f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb29806f7d0 con 0x7fb284004d40 2024-05-09T05:00:18.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb29fb7f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb29806f650 con 0x7fb298162b80 2024-05-09T05:00:18.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb29fb7f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb2980d33f0 con 0x7fb29809ef20 2024-05-09T05:00:18.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb29d91b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb284004d40 0x7fb2980ac760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:18.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb29d91b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb284004d40 0x7fb2980ac760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40546/0 (socket says 172.21.15.60:40546) 2024-05-09T05:00:18.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb29d91b640 1 -- 172.21.15.60:0/1639277590 learned_addr learned my addr 172.21.15.60:0/1639277590 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:18.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb29e11c640 1 --2- 172.21.15.60:0/1639277590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb298162b80 0x7fb298163000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:18.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb29d11a640 1 --2- 172.21.15.60:0/1639277590 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb29809ef20 0x7fb2981f8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:18.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb29d91b640 1 -- 172.21.15.60:0/1639277590 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb29809ef20 msgr2=0x7fb2981f8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:18.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb29d91b640 1 --2- 172.21.15.60:0/1639277590 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb29809ef20 0x7fb2981f8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb29d91b640 1 -- 172.21.15.60:0/1639277590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb298162b80 msgr2=0x7fb298163000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:18.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb29d91b640 1 --2- 172.21.15.60:0/1639277590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb298162b80 0x7fb298163000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb29d91b640 1 -- 172.21.15.60:0/1639277590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2980acca0 con 0x7fb284004d40 2024-05-09T05:00:18.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb29d91b640 1 --2- 172.21.15.60:0/1639277590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb284004d40 0x7fb2980ac760 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb294007340 tx=0x7fb294007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:18.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb28e7fc640 1 -- 172.21.15.60:0/1639277590 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29401a070 con 0x7fb284004d40 2024-05-09T05:00:18.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb28e7fc640 1 -- 172.21.15.60:0/1639277590 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb29400f8c0 con 0x7fb284004d40 2024-05-09T05:00:18.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb29fb7f640 1 -- 172.21.15.60:0/1639277590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2981f93e0 con 0x7fb284004d40 2024-05-09T05:00:18.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.996+0000 7fb28e7fc640 1 -- 172.21.15.60:0/1639277590 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29400fc90 con 0x7fb284004d40 2024-05-09T05:00:18.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.997+0000 7fb29fb7f640 1 -- 172.21.15.60:0/1639277590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2981f99a0 con 0x7fb284004d40 2024-05-09T05:00:18.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.997+0000 7fb29fb7f640 1 -- 172.21.15.60:0/1639277590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb298051040 con 0x7fb284004d40 2024-05-09T05:00:18.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.998+0000 7fb28e7fc640 1 -- 172.21.15.60:0/1639277590 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb29401d070 con 0x7fb284004d40 2024-05-09T05:00:18.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.999+0000 7fb28e7fc640 1 --2- 172.21.15.60:0/1639277590 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb27c076300 0x7fb27c0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:18.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.999+0000 7fb28e7fc640 1 -- 172.21.15.60:0/1639277590 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb2940971f0 con 0x7fb284004d40 2024-05-09T05:00:18.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.999+0000 7fb29d11a640 1 --2- 172.21.15.60:0/1639277590 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb27c076300 0x7fb27c0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:18.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:17.999+0000 7fb29d11a640 1 --2- 172.21.15.60:0/1639277590 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb27c076300 0x7fb27c0787c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb2981639a0 tx=0x7fb288008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:18.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.003+0000 7fb28e7fc640 1 -- 172.21.15.60:0/1639277590 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb294060810 con 0x7fb284004d40 2024-05-09T05:00:18.076 INFO:tasks.ceph.mds.c.smithi060.stderr:2024-05-09T05:00:18.075+0000 7fc7ec960640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T05:00:18.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.233+0000 7fb29fb7f640 1 -- 172.21.15.60:0/1639277590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb29809f8e0 con 0x7fb284004d40 2024-05-09T05:00:18.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.233+0000 7fb28e7fc640 1 -- 172.21.15.60:0/1639277590 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5183 (secure 0 0 0) 0x7fb294060110 con 0x7fb284004d40 2024-05-09T05:00:18.234 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:18.234 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1285340937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1285340937},{"type":"v1","addr":"172.21.15.60:6833","nonce":1285340937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2599193390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2599193390},{"type":"v1","addr":"172.21.15.114:6835","nonce":2599193390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3432609307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3432609307},{"type":"v1","addr":"172.21.15.114:6837","nonce":3432609307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:00:12.208556+0000","modified":"2024-05-09T05:00:17.249167+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2135423314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2135423314},{"type":"v1","addr":"172.21.15.60:6835","nonce":2135423314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-05-09T05:00:18.235 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 7 2024-05-09T05:00:18.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.236+0000 7fb29fb7f640 1 -- 172.21.15.60:0/1639277590 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb27c076300 msgr2=0x7fb27c0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:18.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.236+0000 7fb29fb7f640 1 --2- 172.21.15.60:0/1639277590 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb27c076300 0x7fb27c0787c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb2981639a0 tx=0x7fb288008040 comp rx=0 tx=0).stop 2024-05-09T05:00:18.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.236+0000 7fb29fb7f640 1 -- 172.21.15.60:0/1639277590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb284004d40 msgr2=0x7fb2980ac760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:18.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.236+0000 7fb29fb7f640 1 --2- 172.21.15.60:0/1639277590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb284004d40 0x7fb2980ac760 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb294007340 tx=0x7fb294007800 comp rx=0 tx=0).stop 2024-05-09T05:00:18.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.236+0000 7fb29fb7f640 1 -- 172.21.15.60:0/1639277590 shutdown_connections 2024-05-09T05:00:18.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.236+0000 7fb29fb7f640 1 --2- 172.21.15.60:0/1639277590 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb27c076300 0x7fb27c0787c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.236+0000 7fb29fb7f640 1 --2- 172.21.15.60:0/1639277590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb298162b80 0x7fb298163000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.236+0000 7fb29fb7f640 1 --2- 172.21.15.60:0/1639277590 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb29809ef20 0x7fb2981f8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.236+0000 7fb29fb7f640 1 --2- 172.21.15.60:0/1639277590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb284004d40 0x7fb2980ac760 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.236+0000 7fb29fb7f640 1 -- 172.21.15.60:0/1639277590 >> 172.21.15.60:0/1639277590 conn(0x7fb29809b850 msgr2=0x7fb29815e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:18.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.236+0000 7fb29fb7f640 1 -- 172.21.15.60:0/1639277590 shutdown_connections 2024-05-09T05:00:18.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.236+0000 7fb29fb7f640 1 -- 172.21.15.60:0/1639277590 wait complete. 2024-05-09T05:00:18.247 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:00:18.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.349+0000 7f5857472640 1 Processor -- start 2024-05-09T05:00:18.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.350+0000 7f5857472640 1 -- start start 2024-05-09T05:00:18.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.350+0000 7f5857472640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f585015f300 0x7f58501639f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:18.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.350+0000 7f5857472640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f585006f9f0 0x7f58500d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:18.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.350+0000 7f5857472640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f585006ff60 con 0x7f58500d3930 2024-05-09T05:00:18.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.350+0000 7f5857472640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f5850070240 con 0x7f585015f300 2024-05-09T05:00:18.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.350+0000 7f5857472640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f5850070520 con 0x7f585006f9f0 2024-05-09T05:00:18.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.351+0000 7f585520e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f585015f300 0x7f58501639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:18.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.351+0000 7f585520e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f585015f300 0x7f58501639f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40558/0 (socket says 172.21.15.60:40558) 2024-05-09T05:00:18.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.351+0000 7f585520e640 1 -- 172.21.15.60:0/2155279099 learned_addr learned my addr 172.21.15.60:0/2155279099 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:18.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.351+0000 7f5854a0d640 1 --2- 172.21.15.60:0/2155279099 >> v2:172.21.15.114:3300/0 conn(0x7f585006f9f0 0x7f58500d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:18.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.351+0000 7f585520e640 1 -- 172.21.15.60:0/2155279099 >> v2:172.21.15.114:3300/0 conn(0x7f585006f9f0 msgr2=0x7f58500d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:18.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.351+0000 7f585520e640 1 --2- 172.21.15.60:0/2155279099 >> v2:172.21.15.114:3300/0 conn(0x7f585006f9f0 0x7f58500d33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.351+0000 7f585520e640 1 -- 172.21.15.60:0/2155279099 >> v1:172.21.15.114:6789/0 conn(0x7f58500d3930 legacy=0x7f585006ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:18.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.351+0000 7f585520e640 1 -- 172.21.15.60:0/2155279099 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f585006e740 con 0x7f585015f300 2024-05-09T05:00:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.352+0000 7f585520e640 1 --2- 172.21.15.60:0/2155279099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f585015f300 0x7f58501639f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f585009f890 tx=0x7f584c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb7f2f1547de6a3b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.352+0000 7f5847fff640 1 -- 172.21.15.60:0/2155279099 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584c037070 con 0x7f585015f300 2024-05-09T05:00:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.352+0000 7f5847fff640 1 -- 172.21.15.60:0/2155279099 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f584c032440 con 0x7f585015f300 2024-05-09T05:00:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.352+0000 7f5847fff640 1 -- 172.21.15.60:0/2155279099 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584c032810 con 0x7f585015f300 2024-05-09T05:00:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.352+0000 7f5857472640 1 -- 172.21.15.60:0/2155279099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f585015f300 msgr2=0x7f58501639f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.352+0000 7f5857472640 1 --2- 172.21.15.60:0/2155279099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f585015f300 0x7f58501639f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f585009f890 tx=0x7f584c02a270 comp rx=0 tx=0).stop 2024-05-09T05:00:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.352+0000 7f5857472640 1 -- 172.21.15.60:0/2155279099 shutdown_connections 2024-05-09T05:00:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.352+0000 7f5857472640 1 --2- 172.21.15.60:0/2155279099 >> v2:172.21.15.114:3300/0 conn(0x7f585006f9f0 0x7f58500d33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.352+0000 7f5857472640 1 --2- 172.21.15.60:0/2155279099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f585015f300 0x7f58501639f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.352+0000 7f5857472640 1 -- 172.21.15.60:0/2155279099 >> 172.21.15.60:0/2155279099 conn(0x7f585009b850 msgr2=0x7f585009bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.353+0000 7f5857472640 1 -- 172.21.15.60:0/2155279099 shutdown_connections 2024-05-09T05:00:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.353+0000 7f5857472640 1 -- 172.21.15.60:0/2155279099 wait complete. 2024-05-09T05:00:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.354+0000 7f5857472640 1 Processor -- start 2024-05-09T05:00:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.354+0000 7f5857472640 1 -- start start 2024-05-09T05:00:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.354+0000 7f5857472640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f585006f9f0 0x7f58500ac640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.354+0000 7f5857472640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58500d3930 0x7f5850099a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5857472640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f585015f300 0x7f5850155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5857472640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f585006f650 con 0x7f58500d3930 2024-05-09T05:00:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5857472640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5850163f30 con 0x7f585015f300 2024-05-09T05:00:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5857472640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f585006f7d0 con 0x7f585006f9f0 2024-05-09T05:00:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5854a0d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58500d3930 0x7f5850099a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5854a0d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58500d3930 0x7f5850099a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40562/0 (socket says 172.21.15.60:40562) 2024-05-09T05:00:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5854a0d640 1 -- 172.21.15.60:0/1521400623 learned_addr learned my addr 172.21.15.60:0/1521400623 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f585520e640 1 --2- 172.21.15.60:0/1521400623 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f585006f9f0 0x7f58500ac640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5855a0f640 1 --2- 172.21.15.60:0/1521400623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f585015f300 0x7f5850155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5854a0d640 1 -- 172.21.15.60:0/1521400623 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f585006f9f0 msgr2=0x7f58500ac640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5854a0d640 1 --2- 172.21.15.60:0/1521400623 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f585006f9f0 0x7f58500ac640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5854a0d640 1 -- 172.21.15.60:0/1521400623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f585015f300 msgr2=0x7f5850155f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5854a0d640 1 --2- 172.21.15.60:0/1521400623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f585015f300 0x7f5850155f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5854a0d640 1 -- 172.21.15.60:0/1521400623 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f585009a2d0 con 0x7f58500d3930 2024-05-09T05:00:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5855a0f640 1 --2- 172.21.15.60:0/1521400623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f585015f300 0x7f5850155f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5854a0d640 1 --2- 172.21.15.60:0/1521400623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58500d3930 0x7f5850099a80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f584000e9e0 tx=0x7f584000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5845ffb640 1 -- 172.21.15.60:0/1521400623 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5840017070 con 0x7f58500d3930 2024-05-09T05:00:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5845ffb640 1 -- 172.21.15.60:0/1521400623 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f584000c9d0 con 0x7f58500d3930 2024-05-09T05:00:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5845ffb640 1 -- 172.21.15.60:0/1521400623 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584000cdc0 con 0x7f58500d3930 2024-05-09T05:00:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5857472640 1 -- 172.21.15.60:0/1521400623 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f585009a5b0 con 0x7f58500d3930 2024-05-09T05:00:18.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.355+0000 7f5857472640 1 -- 172.21.15.60:0/1521400623 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5850099fc0 con 0x7f58500d3930 2024-05-09T05:00:18.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.357+0000 7f5857472640 1 -- 172.21.15.60:0/1521400623 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5850051040 con 0x7f58500d3930 2024-05-09T05:00:18.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.357+0000 7f5845ffb640 1 -- 172.21.15.60:0/1521400623 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f584001a070 con 0x7f58500d3930 2024-05-09T05:00:18.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.358+0000 7f5845ffb640 1 --2- 172.21.15.60:0/1521400623 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5834076300 0x7f58340787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:18.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.358+0000 7f5845ffb640 1 -- 172.21.15.60:0/1521400623 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5840097c40 con 0x7f58500d3930 2024-05-09T05:00:18.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.358+0000 7f585520e640 1 --2- 172.21.15.60:0/1521400623 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5834076300 0x7f58340787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:18.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.358+0000 7f585520e640 1 --2- 172.21.15.60:0/1521400623 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5834076300 0x7f58340787c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f584c034000 tx=0x7f584c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:18.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.362+0000 7f5845ffb640 1 -- 172.21.15.60:0/1521400623 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5840060c10 con 0x7f58500d3930 2024-05-09T05:00:18.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.556+0000 7f5857472640 1 -- 172.21.15.60:0/1521400623 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f585009f8e0 con 0x7f58500d3930 2024-05-09T05:00:18.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.557+0000 7f5845ffb640 1 -- 172.21.15.60:0/1521400623 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18490 (secure 0 0 0) 0x7f5840064f10 con 0x7f58500d3930 2024-05-09T05:00:18.558 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:18.558 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:00:17.248958+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:00:10.358133+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:00:11.157769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T05:00:14.850459+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:00:18.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.559+0000 7f5857472640 1 -- 172.21.15.60:0/1521400623 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5834076300 msgr2=0x7f58340787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:18.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.559+0000 7f5857472640 1 --2- 172.21.15.60:0/1521400623 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5834076300 0x7f58340787c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f584c034000 tx=0x7f584c02e000 comp rx=0 tx=0).stop 2024-05-09T05:00:18.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.559+0000 7f5857472640 1 -- 172.21.15.60:0/1521400623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58500d3930 msgr2=0x7f5850099a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:18.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.559+0000 7f5857472640 1 --2- 172.21.15.60:0/1521400623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58500d3930 0x7f5850099a80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f584000e9e0 tx=0x7f584000eea0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.559+0000 7f5857472640 1 -- 172.21.15.60:0/1521400623 shutdown_connections 2024-05-09T05:00:18.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.559+0000 7f5857472640 1 --2- 172.21.15.60:0/1521400623 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5834076300 0x7f58340787c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.559+0000 7f5857472640 1 --2- 172.21.15.60:0/1521400623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f585015f300 0x7f5850155f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.559+0000 7f5857472640 1 --2- 172.21.15.60:0/1521400623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58500d3930 0x7f5850099a80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.559+0000 7f5857472640 1 --2- 172.21.15.60:0/1521400623 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f585006f9f0 0x7f58500ac640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.559+0000 7f5857472640 1 -- 172.21.15.60:0/1521400623 >> 172.21.15.60:0/1521400623 conn(0x7f585009b850 msgr2=0x7f58500d3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:18.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.559+0000 7f5857472640 1 -- 172.21.15.60:0/1521400623 shutdown_connections 2024-05-09T05:00:18.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.559+0000 7f5857472640 1 -- 172.21.15.60:0/1521400623 wait complete. 2024-05-09T05:00:18.571 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:00:18.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.675+0000 7f8c2fb0a640 1 Processor -- start 2024-05-09T05:00:18.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.675+0000 7f8c2fb0a640 1 -- start start 2024-05-09T05:00:18.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.676+0000 7f8c2fb0a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8c2815f300 0x7f8c28072220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:18.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.676+0000 7f8c2fb0a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8c2806fe20 0x7f8c280702d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:18.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.676+0000 7f8c2fb0a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c280736d0 0x7f8c28073b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:18.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.676+0000 7f8c2fb0a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f8c28073240 con 0x7f8c2815f300 2024-05-09T05:00:18.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.676+0000 7f8c2fb0a640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f8c28072760 con 0x7f8c2806fe20 2024-05-09T05:00:18.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.676+0000 7f8c2fb0a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8c280d08e0 con 0x7f8c280736d0 2024-05-09T05:00:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.676+0000 7f8c2d8a6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8c2815f300 0x7f8c28072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.676+0000 7f8c2d8a6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8c2815f300 0x7f8c28072220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35604/0 (socket says 172.21.15.60:35604) 2024-05-09T05:00:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.676+0000 7f8c2d8a6640 1 -- 172.21.15.60:0/4164100638 learned_addr learned my addr 172.21.15.60:0/4164100638 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.676+0000 7f8c2d0a5640 1 --2- 172.21.15.60:0/4164100638 >> v2:172.21.15.114:3300/0 conn(0x7f8c2806fe20 0x7f8c280702d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.676+0000 7f8c2e0a7640 1 --2- 172.21.15.60:0/4164100638 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c280736d0 0x7f8c28073b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.676+0000 7f8c2d8a6640 1 -- 172.21.15.60:0/4164100638 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c280736d0 msgr2=0x7f8c28073b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.676+0000 7f8c2d8a6640 1 --2- 172.21.15.60:0/4164100638 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c280736d0 0x7f8c28073b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.676+0000 7f8c2d8a6640 1 -- 172.21.15.60:0/4164100638 >> v2:172.21.15.114:3300/0 conn(0x7f8c2806fe20 msgr2=0x7f8c280702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.676+0000 7f8c2d8a6640 1 --2- 172.21.15.60:0/4164100638 >> v2:172.21.15.114:3300/0 conn(0x7f8c2806fe20 0x7f8c280702d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.676+0000 7f8c2d8a6640 1 -- 172.21.15.60:0/4164100638 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c280d0bc0 con 0x7f8c2815f300 2024-05-09T05:00:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.677+0000 7f8c2d8a6640 1 --2- 172.21.15.60:0/4164100638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c2815f300 0x7f8c28072220 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8c28071190 tx=0x7f8c2002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24dd0d76c5ca30dd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.677+0000 7f8c2c8a4640 1 -- 172.21.15.60:0/4164100638 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c20037070 con 0x7f8c2815f300 2024-05-09T05:00:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.677+0000 7f8c2c8a4640 1 -- 172.21.15.60:0/4164100638 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c20032440 con 0x7f8c2815f300 2024-05-09T05:00:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.677+0000 7f8c2c8a4640 1 -- 172.21.15.60:0/4164100638 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c20032810 con 0x7f8c2815f300 2024-05-09T05:00:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.678+0000 7f8c2fb0a640 1 -- 172.21.15.60:0/4164100638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c2815f300 msgr2=0x7f8c28072220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.678+0000 7f8c2fb0a640 1 --2- 172.21.15.60:0/4164100638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c2815f300 0x7f8c28072220 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8c28071190 tx=0x7f8c2002a1e0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.678+0000 7f8c2fb0a640 1 -- 172.21.15.60:0/4164100638 shutdown_connections 2024-05-09T05:00:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.678+0000 7f8c2fb0a640 1 --2- 172.21.15.60:0/4164100638 >> v2:172.21.15.114:3300/0 conn(0x7f8c2806fe20 0x7f8c280702d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.678+0000 7f8c2fb0a640 1 --2- 172.21.15.60:0/4164100638 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c280736d0 0x7f8c28073b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.678+0000 7f8c2fb0a640 1 --2- 172.21.15.60:0/4164100638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c2815f300 0x7f8c28072220 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.678+0000 7f8c2fb0a640 1 -- 172.21.15.60:0/4164100638 >> 172.21.15.60:0/4164100638 conn(0x7f8c280b4ef0 msgr2=0x7f8c2806e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.678+0000 7f8c2fb0a640 1 -- 172.21.15.60:0/4164100638 shutdown_connections 2024-05-09T05:00:18.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.678+0000 7f8c2fb0a640 1 -- 172.21.15.60:0/4164100638 wait complete. 2024-05-09T05:00:18.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.679+0000 7f8c2fb0a640 1 Processor -- start 2024-05-09T05:00:18.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.679+0000 7f8c2fb0a640 1 -- start start 2024-05-09T05:00:18.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.680+0000 7f8c2fb0a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c2806fe20 0x7f8c280b1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:18.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.680+0000 7f8c2fb0a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c280736d0 0x7f8c280cfe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:18.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.680+0000 7f8c2fb0a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c2815f300 0x7f8c2815c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.680+0000 7f8c2fb0a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2809c1e0 con 0x7f8c2815f300 2024-05-09T05:00:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.680+0000 7f8c2fb0a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8c280740a0 con 0x7f8c2806fe20 2024-05-09T05:00:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.680+0000 7f8c2fb0a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8c280d04f0 con 0x7f8c280736d0 2024-05-09T05:00:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.680+0000 7f8c2e0a7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c2815f300 0x7f8c2815c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.680+0000 7f8c2e0a7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c2815f300 0x7f8c2815c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35616/0 (socket says 172.21.15.60:35616) 2024-05-09T05:00:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.680+0000 7f8c2e0a7640 1 -- 172.21.15.60:0/1968973813 learned_addr learned my addr 172.21.15.60:0/1968973813 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.680+0000 7f8c2d8a6640 1 --2- 172.21.15.60:0/1968973813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c2806fe20 0x7f8c280b1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.680+0000 7f8c2d0a5640 1 --2- 172.21.15.60:0/1968973813 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c280736d0 0x7f8c280cfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.680+0000 7f8c2e0a7640 1 -- 172.21.15.60:0/1968973813 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c280736d0 msgr2=0x7f8c280cfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.680+0000 7f8c2e0a7640 1 --2- 172.21.15.60:0/1968973813 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c280736d0 0x7f8c280cfe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.680+0000 7f8c2e0a7640 1 -- 172.21.15.60:0/1968973813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c2806fe20 msgr2=0x7f8c280b1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:18.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.680+0000 7f8c2e0a7640 1 --2- 172.21.15.60:0/1968973813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c2806fe20 0x7f8c280b1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.680+0000 7f8c2e0a7640 1 -- 172.21.15.60:0/1968973813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c281f6200 con 0x7f8c2815f300 2024-05-09T05:00:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.680+0000 7f8c2d0a5640 1 --2- 172.21.15.60:0/1968973813 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c280736d0 0x7f8c280cfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.681+0000 7f8c2e0a7640 1 --2- 172.21.15.60:0/1968973813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c2815f300 0x7f8c2815c960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800efc0 tx=0x7f8c1800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.681+0000 7f8c267fc640 1 -- 172.21.15.60:0/1968973813 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c18018020 con 0x7f8c2815f300 2024-05-09T05:00:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.681+0000 7f8c267fc640 1 -- 172.21.15.60:0/1968973813 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c18009b10 con 0x7f8c2815f300 2024-05-09T05:00:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.681+0000 7f8c267fc640 1 -- 172.21.15.60:0/1968973813 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c180146c0 con 0x7f8c2815f300 2024-05-09T05:00:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.681+0000 7f8c2fb0a640 1 -- 172.21.15.60:0/1968973813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c280b21d0 con 0x7f8c2815f300 2024-05-09T05:00:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.681+0000 7f8c2fb0a640 1 -- 172.21.15.60:0/1968973813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c281f6750 con 0x7f8c2815f300 2024-05-09T05:00:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.682+0000 7f8c2fb0a640 1 -- 172.21.15.60:0/1968973813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c28051040 con 0x7f8c2815f300 2024-05-09T05:00:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.682+0000 7f8c267fc640 1 -- 172.21.15.60:0/1968973813 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8c18005d10 con 0x7f8c2815f300 2024-05-09T05:00:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.683+0000 7f8c267fc640 1 --2- 172.21.15.60:0/1968973813 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8c140762d0 0x7f8c14078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.683+0000 7f8c267fc640 1 -- 172.21.15.60:0/1968973813 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f8c180982d0 con 0x7f8c2815f300 2024-05-09T05:00:18.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.683+0000 7f8c2d8a6640 1 --2- 172.21.15.60:0/1968973813 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8c140762d0 0x7f8c14078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:18.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.683+0000 7f8c2d8a6640 1 --2- 172.21.15.60:0/1968973813 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8c140762d0 0x7f8c14078790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8c20034000 tx=0x7f8c2002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:18.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.687+0000 7f8c267fc640 1 -- 172.21.15.60:0/1968973813 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8c18022070 con 0x7f8c2815f300 2024-05-09T05:00:18.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.879+0000 7f8c2fb0a640 1 -- 172.21.15.60:0/1968973813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8c280d4fd0 con 0x7f8c2815f300 2024-05-09T05:00:18.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.880+0000 7f8c267fc640 1 -- 172.21.15.60:0/1968973813 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18490 (secure 0 0 0) 0x7f8c18010040 con 0x7f8c2815f300 2024-05-09T05:00:18.881 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:18.882 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:00:17.248958+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:00:10.358133+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:00:11.157769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T05:00:14.850459+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:00:18.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.882+0000 7f8c2fb0a640 1 -- 172.21.15.60:0/1968973813 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8c140762d0 msgr2=0x7f8c14078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:18.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.882+0000 7f8c2fb0a640 1 --2- 172.21.15.60:0/1968973813 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8c140762d0 0x7f8c14078790 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8c20034000 tx=0x7f8c2002e000 comp rx=0 tx=0).stop 2024-05-09T05:00:18.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.882+0000 7f8c2fb0a640 1 -- 172.21.15.60:0/1968973813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c2815f300 msgr2=0x7f8c2815c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:18.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.882+0000 7f8c2fb0a640 1 --2- 172.21.15.60:0/1968973813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c2815f300 0x7f8c2815c960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800efc0 tx=0x7f8c1800c490 comp rx=0 tx=0).stop 2024-05-09T05:00:18.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.882+0000 7f8c2fb0a640 1 -- 172.21.15.60:0/1968973813 shutdown_connections 2024-05-09T05:00:18.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.882+0000 7f8c2fb0a640 1 --2- 172.21.15.60:0/1968973813 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8c140762d0 0x7f8c14078790 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.882+0000 7f8c2fb0a640 1 --2- 172.21.15.60:0/1968973813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c2815f300 0x7f8c2815c960 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.882+0000 7f8c2fb0a640 1 --2- 172.21.15.60:0/1968973813 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c280736d0 0x7f8c280cfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.883+0000 7f8c2fb0a640 1 --2- 172.21.15.60:0/1968973813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c2806fe20 0x7f8c280b1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:18.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.883+0000 7f8c2fb0a640 1 -- 172.21.15.60:0/1968973813 >> 172.21.15.60:0/1968973813 conn(0x7f8c280b4ef0 msgr2=0x7f8c28162460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:18.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.883+0000 7f8c2fb0a640 1 -- 172.21.15.60:0/1968973813 shutdown_connections 2024-05-09T05:00:18.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:18.883+0000 7f8c2fb0a640 1 -- 172.21.15.60:0/1968973813 wait complete. 2024-05-09T05:00:18.913 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:00:19.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.016+0000 7fe780b46640 1 Processor -- start 2024-05-09T05:00:19.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.016+0000 7fe780b46640 1 -- start start 2024-05-09T05:00:19.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.017+0000 7fe780b46640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe77c1606f0 0x7fe77c0d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:19.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.017+0000 7fe780b46640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe77c06f9f0 0x7fe77c0d3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:19.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.017+0000 7fe780b46640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe77c06ff60 con 0x7fe77c15f300 2024-05-09T05:00:19.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.017+0000 7fe780b46640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fe77c070240 con 0x7fe77c06f9f0 2024-05-09T05:00:19.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.017+0000 7fe780b46640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fe77c070520 con 0x7fe77c1606f0 2024-05-09T05:00:19.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.017+0000 7fe77a59c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe77c15f300 0x7fe77c1639f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44792/0 (socket says 172.21.15.60:44792) 2024-05-09T05:00:19.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.017+0000 7fe77a59c640 1 -- 172.21.15.60:0/1910963548 learned_addr learned my addr 172.21.15.60:0/1910963548 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:19.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.017+0000 7fe77ad9d640 1 --2- 172.21.15.60:0/1910963548 >> v2:172.21.15.60:3300/0 conn(0x7fe77c06f9f0 0x7fe77c0d3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:19.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.017+0000 7fe779d9b640 1 --2- 172.21.15.60:0/1910963548 >> v2:172.21.15.114:3300/0 conn(0x7fe77c1606f0 0x7fe77c0d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:19.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.017+0000 7fe77ad9d640 1 -- 172.21.15.60:0/1910963548 >> v2:172.21.15.114:3300/0 conn(0x7fe77c1606f0 msgr2=0x7fe77c0d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:19.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.017+0000 7fe77ad9d640 1 --2- 172.21.15.60:0/1910963548 >> v2:172.21.15.114:3300/0 conn(0x7fe77c1606f0 0x7fe77c0d33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:19.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.017+0000 7fe77ad9d640 1 -- 172.21.15.60:0/1910963548 >> v1:172.21.15.60:6789/0 conn(0x7fe77c15f300 legacy=0x7fe77c1639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:19.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.017+0000 7fe77ad9d640 1 -- 172.21.15.60:0/1910963548 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe77c06e740 con 0x7fe77c06f9f0 2024-05-09T05:00:19.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.018+0000 7fe77ad9d640 1 --2- 172.21.15.60:0/1910963548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe77c06f9f0 0x7fe77c0d3930 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe774009d90 tx=0x7fe77402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bde2e0f9fcd99fe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:19.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.018+0000 7fe77959a640 1 -- 172.21.15.60:0/1910963548 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe774007740 con 0x7fe77c06f9f0 2024-05-09T05:00:19.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.018+0000 7fe77959a640 1 -- 172.21.15.60:0/1910963548 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7740078e0 con 0x7fe77c06f9f0 2024-05-09T05:00:19.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.018+0000 7fe77959a640 1 -- 172.21.15.60:0/1910963548 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe774007cb0 con 0x7fe77c06f9f0 2024-05-09T05:00:19.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.019+0000 7fe780b46640 1 -- 172.21.15.60:0/1910963548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe77c06f9f0 msgr2=0x7fe77c0d3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:19.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.019+0000 7fe780b46640 1 --2- 172.21.15.60:0/1910963548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe77c06f9f0 0x7fe77c0d3930 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe774009d90 tx=0x7fe77402a6a0 comp rx=0 tx=0).stop 2024-05-09T05:00:19.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.019+0000 7fe780b46640 1 -- 172.21.15.60:0/1910963548 shutdown_connections 2024-05-09T05:00:19.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.019+0000 7fe780b46640 1 --2- 172.21.15.60:0/1910963548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe77c06f9f0 0x7fe77c0d3930 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:19.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.019+0000 7fe780b46640 1 --2- 172.21.15.60:0/1910963548 >> v2:172.21.15.114:3300/0 conn(0x7fe77c1606f0 0x7fe77c0d33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:19.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.019+0000 7fe780b46640 1 -- 172.21.15.60:0/1910963548 >> 172.21.15.60:0/1910963548 conn(0x7fe77c09b850 msgr2=0x7fe77c09bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:19.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.019+0000 7fe780b46640 1 -- 172.21.15.60:0/1910963548 shutdown_connections 2024-05-09T05:00:19.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.019+0000 7fe780b46640 1 -- 172.21.15.60:0/1910963548 wait complete. 2024-05-09T05:00:19.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.020+0000 7fe780b46640 1 Processor -- start 2024-05-09T05:00:19.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.020+0000 7fe780b46640 1 -- start start 2024-05-09T05:00:19.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.021+0000 7fe780b46640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe77c06f9f0 0x7fe77c15a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:19.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.021+0000 7fe780b46640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe77c15f300 0x7fe77c0b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:19.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.021+0000 7fe780b46640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe77c1606f0 0x7fe77c0add50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:19.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.021+0000 7fe780b46640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe77c06f0c0 con 0x7fe77c06f9f0 2024-05-09T05:00:19.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.021+0000 7fe780b46640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe77c06ef40 con 0x7fe77c15f300 2024-05-09T05:00:19.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.021+0000 7fe780b46640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe77c06f240 con 0x7fe77c1606f0 2024-05-09T05:00:19.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.021+0000 7fe77a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe77c06f9f0 0x7fe77c15a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:19.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.021+0000 7fe77a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe77c06f9f0 0x7fe77c15a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35634/0 (socket says 172.21.15.60:35634) 2024-05-09T05:00:19.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.021+0000 7fe77a59c640 1 -- 172.21.15.60:0/1837970935 learned_addr learned my addr 172.21.15.60:0/1837970935 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:19.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.021+0000 7fe779d9b640 1 --2- 172.21.15.60:0/1837970935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe77c15f300 0x7fe77c0b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:19.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.021+0000 7fe77ad9d640 1 --2- 172.21.15.60:0/1837970935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe77c1606f0 0x7fe77c0add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:19.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.021+0000 7fe77a59c640 1 -- 172.21.15.60:0/1837970935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe77c1606f0 msgr2=0x7fe77c0add50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:19.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.021+0000 7fe77a59c640 1 --2- 172.21.15.60:0/1837970935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe77c1606f0 0x7fe77c0add50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:19.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.021+0000 7fe77a59c640 1 -- 172.21.15.60:0/1837970935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe77c15f300 msgr2=0x7fe77c0b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:19.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.021+0000 7fe77a59c640 1 --2- 172.21.15.60:0/1837970935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe77c15f300 0x7fe77c0b1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:19.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.021+0000 7fe77a59c640 1 -- 172.21.15.60:0/1837970935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe77c0b16c0 con 0x7fe77c06f9f0 2024-05-09T05:00:19.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.021+0000 7fe779d9b640 1 --2- 172.21.15.60:0/1837970935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe77c15f300 0x7fe77c0b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:19.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.022+0000 7fe77a59c640 1 --2- 172.21.15.60:0/1837970935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe77c06f9f0 0x7fe77c15a540 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe76c00c4d0 tx=0x7fe76c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:19.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.022+0000 7fe76affd640 1 -- 172.21.15.60:0/1837970935 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76c017070 con 0x7fe77c06f9f0 2024-05-09T05:00:19.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.022+0000 7fe76affd640 1 -- 172.21.15.60:0/1837970935 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe76c005d00 con 0x7fe77c06f9f0 2024-05-09T05:00:19.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.022+0000 7fe780b46640 1 -- 172.21.15.60:0/1837970935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe77c0ae8c0 con 0x7fe77c06f9f0 2024-05-09T05:00:19.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.022+0000 7fe76affd640 1 -- 172.21.15.60:0/1837970935 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76c0072b0 con 0x7fe77c06f9f0 2024-05-09T05:00:19.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.022+0000 7fe780b46640 1 -- 172.21.15.60:0/1837970935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe77c09ca00 con 0x7fe77c06f9f0 2024-05-09T05:00:19.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.023+0000 7fe780b46640 1 -- 172.21.15.60:0/1837970935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe77c051040 con 0x7fe77c06f9f0 2024-05-09T05:00:19.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.023+0000 7fe76affd640 1 -- 172.21.15.60:0/1837970935 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe76c01a070 con 0x7fe77c06f9f0 2024-05-09T05:00:19.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.024+0000 7fe76affd640 1 --2- 172.21.15.60:0/1837970935 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe75c0762d0 0x7fe75c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:19.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.024+0000 7fe76affd640 1 -- 172.21.15.60:0/1837970935 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe76c096c80 con 0x7fe77c06f9f0 2024-05-09T05:00:19.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.024+0000 7fe779d9b640 1 --2- 172.21.15.60:0/1837970935 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe75c0762d0 0x7fe75c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:19.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.025+0000 7fe779d9b640 1 --2- 172.21.15.60:0/1837970935 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe75c0762d0 0x7fe75c078790 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe770006fd0 tx=0x7fe770008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:19.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.028+0000 7fe76affd640 1 -- 172.21.15.60:0/1837970935 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe76c0602a0 con 0x7fe77c06f9f0 2024-05-09T05:00:19.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.255+0000 7fe780b46640 1 -- 172.21.15.60:0/1837970935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe77c09f8e0 con 0x7fe77c06f9f0 2024-05-09T05:00:19.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.255+0000 7fe76affd640 1 -- 172.21.15.60:0/1837970935 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5183 (secure 0 0 0) 0x7fe76c05fba0 con 0x7fe77c06f9f0 2024-05-09T05:00:19.256 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:19.256 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1285340937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1285340937},{"type":"v1","addr":"172.21.15.60:6833","nonce":1285340937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2599193390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2599193390},{"type":"v1","addr":"172.21.15.114:6835","nonce":2599193390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3432609307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3432609307},{"type":"v1","addr":"172.21.15.114:6837","nonce":3432609307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:00:12.208556+0000","modified":"2024-05-09T05:00:17.249167+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2135423314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2135423314},{"type":"v1","addr":"172.21.15.60:6835","nonce":2135423314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-05-09T05:00:19.256 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 7 2024-05-09T05:00:19.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.257+0000 7fe780b46640 1 -- 172.21.15.60:0/1837970935 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe75c0762d0 msgr2=0x7fe75c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:19.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.257+0000 7fe780b46640 1 --2- 172.21.15.60:0/1837970935 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe75c0762d0 0x7fe75c078790 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe770006fd0 tx=0x7fe770008040 comp rx=0 tx=0).stop 2024-05-09T05:00:19.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.257+0000 7fe780b46640 1 -- 172.21.15.60:0/1837970935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe77c06f9f0 msgr2=0x7fe77c15a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:19.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.257+0000 7fe780b46640 1 --2- 172.21.15.60:0/1837970935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe77c06f9f0 0x7fe77c15a540 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe76c00c4d0 tx=0x7fe76c00c990 comp rx=0 tx=0).stop 2024-05-09T05:00:19.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.257+0000 7fe780b46640 1 -- 172.21.15.60:0/1837970935 shutdown_connections 2024-05-09T05:00:19.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.257+0000 7fe780b46640 1 --2- 172.21.15.60:0/1837970935 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe75c0762d0 0x7fe75c078790 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:19.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.257+0000 7fe780b46640 1 --2- 172.21.15.60:0/1837970935 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe77c1606f0 0x7fe77c0add50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:19.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.257+0000 7fe780b46640 1 --2- 172.21.15.60:0/1837970935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe77c15f300 0x7fe77c0b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:19.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.257+0000 7fe780b46640 1 --2- 172.21.15.60:0/1837970935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe77c06f9f0 0x7fe77c15a540 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:19.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.257+0000 7fe780b46640 1 -- 172.21.15.60:0/1837970935 >> 172.21.15.60:0/1837970935 conn(0x7fe77c09b850 msgr2=0x7fe77c0d2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:19.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.257+0000 7fe780b46640 1 -- 172.21.15.60:0/1837970935 shutdown_connections 2024-05-09T05:00:19.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.257+0000 7fe780b46640 1 -- 172.21.15.60:0/1837970935 wait complete. 2024-05-09T05:00:19.269 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:00:19.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.372+0000 7fefd4cd1640 1 Processor -- start 2024-05-09T05:00:19.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.372+0000 7fefd4cd1640 1 -- start start 2024-05-09T05:00:19.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.373+0000 7fefd4cd1640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fefd01606f0 0x7fefd00d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:19.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.373+0000 7fefd4cd1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fefd006f9f0 0x7fefd00d3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:19.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.373+0000 7fefd4cd1640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefd006ff60 con 0x7fefd015f300 2024-05-09T05:00:19.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.373+0000 7fefd4cd1640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fefd0070240 con 0x7fefd01606f0 2024-05-09T05:00:19.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.373+0000 7fefd4cd1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fefd0070520 con 0x7fefd006f9f0 2024-05-09T05:00:19.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.373+0000 7fefce59c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fefd015f300 0x7fefd01639f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44798/0 (socket says 172.21.15.60:44798) 2024-05-09T05:00:19.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.373+0000 7fefce59c640 1 -- 172.21.15.60:0/1729749858 learned_addr learned my addr 172.21.15.60:0/1729749858 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:19.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.373+0000 7fefc7fff640 1 --2- 172.21.15.60:0/1729749858 >> v2:172.21.15.114:3300/0 conn(0x7fefd01606f0 0x7fefd00d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:19.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.373+0000 7fefced9d640 1 --2- 172.21.15.60:0/1729749858 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fefd006f9f0 0x7fefd00d3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:19.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.373+0000 7fefcdd9b640 1 -- 172.21.15.60:0/1729749858 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2170770914 0 0) 0x7fefd006ff60 con 0x7fefd015f300 2024-05-09T05:00:19.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.373+0000 7fefcdd9b640 1 -- 172.21.15.60:0/1729749858 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefbc003520 con 0x7fefd015f300 2024-05-09T05:00:19.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.373+0000 7fefc7fff640 1 -- 172.21.15.60:0/1729749858 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fefd006f9f0 msgr2=0x7fefd00d3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:19.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.373+0000 7fefc7fff640 1 --2- 172.21.15.60:0/1729749858 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fefd006f9f0 0x7fefd00d3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:19.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.373+0000 7fefc7fff640 1 -- 172.21.15.60:0/1729749858 >> v1:172.21.15.60:6789/0 conn(0x7fefd015f300 legacy=0x7fefd01639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:19.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.373+0000 7fefc7fff640 1 -- 172.21.15.60:0/1729749858 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefd006e740 con 0x7fefd01606f0 2024-05-09T05:00:19.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.373+0000 7fefced9d640 1 --2- 172.21.15.60:0/1729749858 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fefd006f9f0 0x7fefd00d3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:00:19.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.374+0000 7fefc7fff640 1 --2- 172.21.15.60:0/1729749858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd01606f0 0x7fefd00d33f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fefb80097a0 tx=0x7fefb802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec19fda40c5201b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:19.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.374+0000 7fefcdd9b640 1 -- 172.21.15.60:0/1729749858 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb8037070 con 0x7fefd01606f0 2024-05-09T05:00:19.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.375+0000 7fefcdd9b640 1 -- 172.21.15.60:0/1729749858 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefb8032440 con 0x7fefd01606f0 2024-05-09T05:00:19.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.375+0000 7fefcdd9b640 1 -- 172.21.15.60:0/1729749858 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb80327f0 con 0x7fefd01606f0 2024-05-09T05:00:19.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.375+0000 7fefd4cd1640 1 -- 172.21.15.60:0/1729749858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd01606f0 msgr2=0x7fefd00d33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:19.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.375+0000 7fefd4cd1640 1 --2- 172.21.15.60:0/1729749858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd01606f0 0x7fefd00d33f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fefb80097a0 tx=0x7fefb802a180 comp rx=0 tx=0).stop 2024-05-09T05:00:19.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.375+0000 7fefd4cd1640 1 -- 172.21.15.60:0/1729749858 shutdown_connections 2024-05-09T05:00:19.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.375+0000 7fefd4cd1640 1 --2- 172.21.15.60:0/1729749858 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fefd006f9f0 0x7fefd00d3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:19.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.375+0000 7fefd4cd1640 1 --2- 172.21.15.60:0/1729749858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd01606f0 0x7fefd00d33f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:19.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.375+0000 7fefd4cd1640 1 -- 172.21.15.60:0/1729749858 >> 172.21.15.60:0/1729749858 conn(0x7fefd009b850 msgr2=0x7fefd009bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:19.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.375+0000 7fefd4cd1640 1 -- 172.21.15.60:0/1729749858 shutdown_connections 2024-05-09T05:00:19.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.375+0000 7fefd4cd1640 1 -- 172.21.15.60:0/1729749858 wait complete. 2024-05-09T05:00:19.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.376+0000 7fefd4cd1640 1 Processor -- start 2024-05-09T05:00:19.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.377+0000 7fefd4cd1640 1 -- start start 2024-05-09T05:00:19.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.377+0000 7fefd4cd1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fefd006f9f0 0x7fefd009cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:19.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.377+0000 7fefd4cd1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd015f300 0x7fefd009eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:19.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.377+0000 7fefd4cd1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefd01606f0 0x7fefd009f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:19.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.377+0000 7fefd4cd1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fefd006f0c0 con 0x7fefd01606f0 2024-05-09T05:00:19.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.377+0000 7fefd4cd1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fefd006ef40 con 0x7fefd015f300 2024-05-09T05:00:19.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.377+0000 7fefd4cd1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fefd006f240 con 0x7fefd006f9f0 2024-05-09T05:00:19.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.377+0000 7fefced9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefd01606f0 0x7fefd009f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:19.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.377+0000 7fefced9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefd01606f0 0x7fefd009f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35642/0 (socket says 172.21.15.60:35642) 2024-05-09T05:00:19.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.377+0000 7fefced9d640 1 -- 172.21.15.60:0/1082518617 learned_addr learned my addr 172.21.15.60:0/1082518617 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:19.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.378+0000 7fefc7fff640 1 --2- 172.21.15.60:0/1082518617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd015f300 0x7fefd009eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:19.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.378+0000 7fefce59c640 1 --2- 172.21.15.60:0/1082518617 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fefd006f9f0 0x7fefd009cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:19.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.378+0000 7fefced9d640 1 -- 172.21.15.60:0/1082518617 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fefd006f9f0 msgr2=0x7fefd009cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:19.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.378+0000 7fefced9d640 1 --2- 172.21.15.60:0/1082518617 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fefd006f9f0 0x7fefd009cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:19.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.378+0000 7fefced9d640 1 -- 172.21.15.60:0/1082518617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd015f300 msgr2=0x7fefd009eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:19.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.378+0000 7fefced9d640 1 --2- 172.21.15.60:0/1082518617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd015f300 0x7fefd009eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:19.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.378+0000 7fefced9d640 1 -- 172.21.15.60:0/1082518617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefd00aff50 con 0x7fefd01606f0 2024-05-09T05:00:19.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.378+0000 7fefce59c640 1 --2- 172.21.15.60:0/1082518617 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fefd006f9f0 0x7fefd009cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:19.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.378+0000 7fefced9d640 1 --2- 172.21.15.60:0/1082518617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefd01606f0 0x7fefd009f060 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fefc8009f00 tx=0x7fefc800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:19.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.378+0000 7fefc6ffd640 1 -- 172.21.15.60:0/1082518617 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc8018030 con 0x7fefd01606f0 2024-05-09T05:00:19.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.378+0000 7fefc6ffd640 1 -- 172.21.15.60:0/1082518617 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefc800ce50 con 0x7fefd01606f0 2024-05-09T05:00:19.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.378+0000 7fefd4cd1640 1 -- 172.21.15.60:0/1082518617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefd009c7d0 con 0x7fefd01606f0 2024-05-09T05:00:19.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.378+0000 7fefc6ffd640 1 -- 172.21.15.60:0/1082518617 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc8003bb0 con 0x7fefd01606f0 2024-05-09T05:00:19.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.378+0000 7fefd4cd1640 1 -- 172.21.15.60:0/1082518617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefd009d450 con 0x7fefd01606f0 2024-05-09T05:00:19.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.379+0000 7fefd4cd1640 1 -- 172.21.15.60:0/1082518617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefd0051040 con 0x7fefd01606f0 2024-05-09T05:00:19.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.380+0000 7fefc6ffd640 1 -- 172.21.15.60:0/1082518617 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fefc801b070 con 0x7fefd01606f0 2024-05-09T05:00:19.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.380+0000 7fefc6ffd640 1 --2- 172.21.15.60:0/1082518617 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fefb00762d0 0x7fefb0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:19.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.380+0000 7fefc6ffd640 1 -- 172.21.15.60:0/1082518617 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fefc8098330 con 0x7fefd01606f0 2024-05-09T05:00:19.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.381+0000 7fefce59c640 1 --2- 172.21.15.60:0/1082518617 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fefb00762d0 0x7fefb0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:19.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.381+0000 7fefce59c640 1 --2- 172.21.15.60:0/1082518617 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fefb00762d0 0x7fefb0078790 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fefc0007fd0 tx=0x7fefc0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:19.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.385+0000 7fefc6ffd640 1 -- 172.21.15.60:0/1082518617 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fefc8010040 con 0x7fefd01606f0 2024-05-09T05:00:19.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.574+0000 7fefd4cd1640 1 -- 172.21.15.60:0/1082518617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fefd009c9a0 con 0x7fefd01606f0 2024-05-09T05:00:19.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.575+0000 7fefc6ffd640 1 -- 172.21.15.60:0/1082518617 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18490 (secure 0 0 0) 0x7fefc80618b0 con 0x7fefd01606f0 2024-05-09T05:00:19.576 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:19.576 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:00:17.248958+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:00:10.358133+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:00:11.157769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T05:00:14.850459+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:00:19.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.578+0000 7fefd4cd1640 1 -- 172.21.15.60:0/1082518617 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fefb00762d0 msgr2=0x7fefb0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:19.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.578+0000 7fefd4cd1640 1 --2- 172.21.15.60:0/1082518617 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fefb00762d0 0x7fefb0078790 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fefc0007fd0 tx=0x7fefc0009040 comp rx=0 tx=0).stop 2024-05-09T05:00:19.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.578+0000 7fefd4cd1640 1 -- 172.21.15.60:0/1082518617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefd01606f0 msgr2=0x7fefd009f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:19.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.578+0000 7fefd4cd1640 1 --2- 172.21.15.60:0/1082518617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefd01606f0 0x7fefd009f060 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fefc8009f00 tx=0x7fefc800c490 comp rx=0 tx=0).stop 2024-05-09T05:00:19.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.578+0000 7fefd4cd1640 1 -- 172.21.15.60:0/1082518617 shutdown_connections 2024-05-09T05:00:19.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.578+0000 7fefd4cd1640 1 --2- 172.21.15.60:0/1082518617 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fefb00762d0 0x7fefb0078790 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:19.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.578+0000 7fefd4cd1640 1 --2- 172.21.15.60:0/1082518617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefd01606f0 0x7fefd009f060 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:19.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.578+0000 7fefd4cd1640 1 --2- 172.21.15.60:0/1082518617 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefd015f300 0x7fefd009eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:19.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.578+0000 7fefd4cd1640 1 --2- 172.21.15.60:0/1082518617 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fefd006f9f0 0x7fefd009cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:19.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.578+0000 7fefd4cd1640 1 -- 172.21.15.60:0/1082518617 >> 172.21.15.60:0/1082518617 conn(0x7fefd009b850 msgr2=0x7fefd00d4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:19.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.578+0000 7fefd4cd1640 1 -- 172.21.15.60:0/1082518617 shutdown_connections 2024-05-09T05:00:19.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:19.578+0000 7fefd4cd1640 1 -- 172.21.15.60:0/1082518617 wait complete. 2024-05-09T05:00:19.992 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T05:00:12.208556+0000', 'modified': '2024-05-09T05:00:17.249167+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4261}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4261': {'gid': 4261, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6835/2135423314', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6834', 'nonce': 2135423314}, {'type': 'v1', 'addr': '172.21.15.60:6835', 'nonce': 2135423314}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4261, 'qdb_cluster': [4261]} 2024-05-09T05:00:19.992 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T05:00:19.992 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T05:00:20.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.101+0000 7f40ab237640 1 Processor -- start 2024-05-09T05:00:20.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.101+0000 7f40ab237640 1 -- start start 2024-05-09T05:00:20.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.101+0000 7f40ab237640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f40a41605d0 0x7f40a40b1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.102+0000 7f40ab237640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40a40b2500 con 0x7f40a41598f0 2024-05-09T05:00:20.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.102+0000 7f40ab237640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40a40b27e0 con 0x7f40a4072130 2024-05-09T05:00:20.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.102+0000 7f40ab237640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f40a40addd0 con 0x7f40a41605d0 2024-05-09T05:00:20.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.102+0000 7f40a3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f40a41605d0 0x7f40a40b1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.102+0000 7f40a8fd3640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f40a41598f0 0x7f40a4160ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44806/0 (socket says 172.21.15.60:44806) 2024-05-09T05:00:20.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.102+0000 7f40a8fd3640 1 -- 172.21.15.60:0/882820543 learned_addr learned my addr 172.21.15.60:0/882820543 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:20.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.102+0000 7f40a3fff640 1 -- 172.21.15.60:0/882820543 >> v1:172.21.15.114:6789/0 conn(0x7f40a4072130 legacy=0x7f40a40ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:20.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.102+0000 7f40a3fff640 1 -- 172.21.15.60:0/882820543 >> v1:172.21.15.60:6789/0 conn(0x7f40a41598f0 legacy=0x7f40a4160ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:20.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.102+0000 7f40a3fff640 1 -- 172.21.15.60:0/882820543 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a40b2ac0 con 0x7f40a41605d0 2024-05-09T05:00:20.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.102+0000 7f40a37fe640 1 -- 172.21.15.60:0/882820543 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 488098194 0 0) 0x7f40a40b2500 con 0x7f40a41598f0 2024-05-09T05:00:20.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.103+0000 7f40a3fff640 1 --2- 172.21.15.60:0/882820543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40a41605d0 0x7f40a40b1cb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f40a40adf50 tx=0x7f409402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed41490dfd450d8c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:20.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.103+0000 7f40a37fe640 1 -- 172.21.15.60:0/882820543 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094031600 con 0x7f40a41605d0 2024-05-09T05:00:20.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.103+0000 7f40a37fe640 1 -- 172.21.15.60:0/882820543 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40940317a0 con 0x7f40a41605d0 2024-05-09T05:00:20.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.103+0000 7f40a37fe640 1 -- 172.21.15.60:0/882820543 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094031b70 con 0x7f40a41605d0 2024-05-09T05:00:20.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.103+0000 7f40ab237640 1 -- 172.21.15.60:0/882820543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40a41605d0 msgr2=0x7f40a40b1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:20.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.103+0000 7f40ab237640 1 --2- 172.21.15.60:0/882820543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40a41605d0 0x7f40a40b1cb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f40a40adf50 tx=0x7f409402a5d0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.103+0000 7f40ab237640 1 -- 172.21.15.60:0/882820543 shutdown_connections 2024-05-09T05:00:20.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.103+0000 7f40ab237640 1 --2- 172.21.15.60:0/882820543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40a41605d0 0x7f40a40b1cb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.103+0000 7f40ab237640 1 -- 172.21.15.60:0/882820543 >> 172.21.15.60:0/882820543 conn(0x7f40a40b4850 msgr2=0x7f40a40b4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:20.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.103+0000 7f40ab237640 1 -- 172.21.15.60:0/882820543 shutdown_connections 2024-05-09T05:00:20.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.103+0000 7f40ab237640 1 -- 172.21.15.60:0/882820543 wait complete. 2024-05-09T05:00:20.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.105+0000 7f40ab237640 1 Processor -- start 2024-05-09T05:00:20.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.105+0000 7f40ab237640 1 -- start start 2024-05-09T05:00:20.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.105+0000 7f40ab237640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40a4072130 0x7f40a41f4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40ab237640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40a41598f0 0x7f40a415f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40ab237640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40a41605d0 0x7f40a415f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40ab237640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f40a4159cd0 con 0x7f40a41598f0 2024-05-09T05:00:20.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40ab237640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f40a40b21f0 con 0x7f40a41605d0 2024-05-09T05:00:20.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40ab237640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f40a415a080 con 0x7f40a4072130 2024-05-09T05:00:20.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40a3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40a41598f0 0x7f40a415f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40a3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40a41598f0 0x7f40a415f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35660/0 (socket says 172.21.15.60:35660) 2024-05-09T05:00:20.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40a3fff640 1 -- 172.21.15.60:0/120006917 learned_addr learned my addr 172.21.15.60:0/120006917 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:20.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40a97d4640 1 --2- 172.21.15.60:0/120006917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40a41605d0 0x7f40a415f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40a8fd3640 1 --2- 172.21.15.60:0/120006917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40a4072130 0x7f40a41f4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40a3fff640 1 -- 172.21.15.60:0/120006917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40a4072130 msgr2=0x7f40a41f4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:20.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40a3fff640 1 --2- 172.21.15.60:0/120006917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40a4072130 0x7f40a41f4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40a3fff640 1 -- 172.21.15.60:0/120006917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40a41605d0 msgr2=0x7f40a415f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:20.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40a3fff640 1 --2- 172.21.15.60:0/120006917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40a41605d0 0x7f40a415f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40a3fff640 1 -- 172.21.15.60:0/120006917 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a415a320 con 0x7f40a41598f0 2024-05-09T05:00:20.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40a8fd3640 1 --2- 172.21.15.60:0/120006917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40a4072130 0x7f40a41f4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:20.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40a97d4640 1 --2- 172.21.15.60:0/120006917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40a41605d0 0x7f40a415f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:20.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40a3fff640 1 --2- 172.21.15.60:0/120006917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40a41598f0 0x7f40a415f480 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4094009e80 tx=0x7f4094004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:20.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40a17fa640 1 -- 172.21.15.60:0/120006917 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094040070 con 0x7f40a41598f0 2024-05-09T05:00:20.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40a17fa640 1 -- 172.21.15.60:0/120006917 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4094030be0 con 0x7f40a41598f0 2024-05-09T05:00:20.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40ab237640 1 -- 172.21.15.60:0/120006917 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40a415a600 con 0x7f40a41598f0 2024-05-09T05:00:20.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40a17fa640 1 -- 172.21.15.60:0/120006917 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094045460 con 0x7f40a41598f0 2024-05-09T05:00:20.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.106+0000 7f40ab237640 1 -- 172.21.15.60:0/120006917 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40a415abc0 con 0x7f40a41598f0 2024-05-09T05:00:20.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.109+0000 7f40a17fa640 1 -- 172.21.15.60:0/120006917 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4094039040 con 0x7f40a41598f0 2024-05-09T05:00:20.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.109+0000 7f40a17fa640 1 --2- 172.21.15.60:0/120006917 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f40840762d0 0x7f4084078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.109+0000 7f40a17fa640 1 -- 172.21.15.60:0/120006917 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f40940b9200 con 0x7f40a41598f0 2024-05-09T05:00:20.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.109+0000 7f40a8fd3640 1 --2- 172.21.15.60:0/120006917 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f40840762d0 0x7f4084078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.110+0000 7f40ae02c740 1 Processor -- start 2024-05-09T05:00:20.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.110+0000 7f40ae02c740 1 -- start start 2024-05-09T05:00:20.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.110+0000 7f40a8fd3640 1 --2- 172.21.15.60:0/120006917 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f40840762d0 0x7f4084078790 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f409c000f60 tx=0x7f409c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:20.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.110+0000 7f40ae02c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fb7ff0 0x562ce5fba800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.110+0000 7f40ae02c740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb83d0 0x562ce5fbad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.110+0000 7f40ae02c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5f8ac50 0x562ce5f8e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.110+0000 7f40ae02c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x562ce5ef48d0 con 0x562ce5f8ac50 2024-05-09T05:00:20.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.110+0000 7f40ae02c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562ce5eadc10 con 0x562ce5fb7ff0 2024-05-09T05:00:20.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.110+0000 7f40ae02c740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x562ce5ef0890 con 0x562ce5fb83d0 2024-05-09T05:00:20.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.110+0000 7f40a97d4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5f8ac50 0x562ce5f8e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.110+0000 7f40a97d4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5f8ac50 0x562ce5f8e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35676/0 (socket says 172.21.15.60:35676) 2024-05-09T05:00:20.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.110+0000 7f40a97d4640 1 -- 172.21.15.60:0/3063207567 learned_addr learned my addr 172.21.15.60:0/3063207567 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:20.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.110+0000 7f40a8fd3640 1 --2- 172.21.15.60:0/3063207567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fb7ff0 0x562ce5fba800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.110+0000 7f40a3fff640 1 --2- 172.21.15.60:0/3063207567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb83d0 0x562ce5fbad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.111+0000 7f40a97d4640 1 -- 172.21.15.60:0/3063207567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb83d0 msgr2=0x562ce5fbad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:20.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.111+0000 7f40a97d4640 1 --2- 172.21.15.60:0/3063207567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb83d0 0x562ce5fbad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.111+0000 7f40a97d4640 1 -- 172.21.15.60:0/3063207567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fb7ff0 msgr2=0x562ce5fba800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:20.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.111+0000 7f40a97d4640 1 --2- 172.21.15.60:0/3063207567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fb7ff0 0x562ce5fba800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.111+0000 7f40a97d4640 1 -- 172.21.15.60:0/3063207567 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ce5eb55d0 con 0x562ce5f8ac50 2024-05-09T05:00:20.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.111+0000 7f40a97d4640 1 --2- 172.21.15.60:0/3063207567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5f8ac50 0x562ce5f8e460 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f408c0016c0 tx=0x7f408c001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ee4b2353554945 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:20.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.111+0000 7f409a7fc640 1 -- 172.21.15.60:0/3063207567 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c018070 con 0x562ce5f8ac50 2024-05-09T05:00:20.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.111+0000 7f409a7fc640 1 -- 172.21.15.60:0/3063207567 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f408c00d6d0 con 0x562ce5f8ac50 2024-05-09T05:00:20.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.111+0000 7f409a7fc640 1 -- 172.21.15.60:0/3063207567 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c00dac0 con 0x562ce5f8ac50 2024-05-09T05:00:20.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.112+0000 7f40ae02c740 1 -- 172.21.15.60:0/3063207567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5f8ac50 msgr2=0x562ce5f8e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:20.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.112+0000 7f40ae02c740 1 --2- 172.21.15.60:0/3063207567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5f8ac50 0x562ce5f8e460 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f408c0016c0 tx=0x7f408c001b80 comp rx=0 tx=0).stop 2024-05-09T05:00:20.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.112+0000 7f40ae02c740 1 -- 172.21.15.60:0/3063207567 shutdown_connections 2024-05-09T05:00:20.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.112+0000 7f40ae02c740 1 --2- 172.21.15.60:0/3063207567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5f8ac50 0x562ce5f8e460 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.112+0000 7f40ae02c740 1 --2- 172.21.15.60:0/3063207567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb83d0 0x562ce5fbad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.112+0000 7f40ae02c740 1 --2- 172.21.15.60:0/3063207567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fb7ff0 0x562ce5fba800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.112+0000 7f40ae02c740 1 -- 172.21.15.60:0/3063207567 >> 172.21.15.60:0/3063207567 conn(0x562ce5f8cec0 msgr2=0x562ce5fd5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:20.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.112+0000 7f40ae02c740 1 -- 172.21.15.60:0/3063207567 shutdown_connections 2024-05-09T05:00:20.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.112+0000 7f40ae02c740 1 -- 172.21.15.60:0/3063207567 wait complete. 2024-05-09T05:00:20.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.112+0000 7f40ae02c740 1 -- start start 2024-05-09T05:00:20.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.112+0000 7f40ae02c740 1 Processor -- start 2024-05-09T05:00:20.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.113+0000 7f40ae02c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fb7ff0 0x562ce5faee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.113+0000 7f40ae02c740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb83d0 0x562ce5f8f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.113+0000 7f40ae02c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fadfd0 0x562ce5f8f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.113+0000 7f40ae02c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x562ce5f90070 con 0x562ce5fadfd0 2024-05-09T05:00:20.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.113+0000 7f40ae02c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562ce5eb1760 con 0x562ce5fb7ff0 2024-05-09T05:00:20.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.113+0000 7f40ae02c740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x562ce5f90350 con 0x562ce5fb83d0 2024-05-09T05:00:20.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.113+0000 7f40ae02c740 10 client.0 _ll_register_callbacks cb 0x562ce5eeacf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:00:20.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.113+0000 7f40a97d4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fadfd0 0x562ce5f8f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.113+0000 7f40a97d4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fadfd0 0x562ce5f8f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35680/0 (socket says 172.21.15.60:35680) 2024-05-09T05:00:20.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.113+0000 7f40a97d4640 1 -- 172.21.15.60:0/212887998 learned_addr learned my addr 172.21.15.60:0/212887998 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:20.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.113+0000 7f40a8fd3640 1 --2- 172.21.15.60:0/212887998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fb7ff0 0x562ce5faee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.113+0000 7f40a3fff640 1 --2- 172.21.15.60:0/212887998 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb83d0 0x562ce5f8f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40a97d4640 1 -- 172.21.15.60:0/212887998 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb83d0 msgr2=0x562ce5f8f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:20.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40a97d4640 1 --2- 172.21.15.60:0/212887998 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb83d0 0x562ce5f8f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40a97d4640 1 -- 172.21.15.60:0/212887998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fb7ff0 msgr2=0x562ce5faee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:20.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40a97d4640 1 --2- 172.21.15.60:0/212887998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fb7ff0 0x562ce5faee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40a97d4640 1 -- 172.21.15.60:0/212887998 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ce5c062d0 con 0x562ce5fadfd0 2024-05-09T05:00:20.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40a97d4640 1 --2- 172.21.15.60:0/212887998 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fadfd0 0x562ce5f8f9b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f408c001bc0 tx=0x7f408c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8dd691fbcce8e11 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:20.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40837fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-05-09T05:00:20.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40837fe640 1 -- 172.21.15.60:0/212887998 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c021020 con 0x562ce5fadfd0 2024-05-09T05:00:20.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40837fe640 1 -- 172.21.15.60:0/212887998 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f408c00db00 con 0x562ce5fadfd0 2024-05-09T05:00:20.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40ae02c740 1 -- 172.21.15.60:0/212887998 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f408c00d6d0 con 0x562ce5fadfd0 2024-05-09T05:00:20.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40837fe640 1 -- 172.21.15.60:0/212887998 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f408c02b050 con 0x562ce5fadfd0 2024-05-09T05:00:20.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40ae02c740 10 client.4392 fetch_fsmap learned FSMap version 7 2024-05-09T05:00:20.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40ae02c740 1 -- 172.21.15.60:0/212887998 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x562ce5cd06b0 con 0x562ce5fadfd0 2024-05-09T05:00:20.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40837fe640 1 -- 172.21.15.60:0/212887998 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f408c00d6d0 con 0x562ce5fadfd0 2024-05-09T05:00:20.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40837fe640 1 -- 172.21.15.60:0/212887998 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7f408c02e020 con 0x562ce5fadfd0 2024-05-09T05:00:20.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40ae02c740 10 client.4392 fetch_fsmap finished waiting for FSMap version 7 2024-05-09T05:00:20.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40ae02c740 10 client.4392 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T05:00:20.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40ae02c740 20 client.4392 populate_metadata read hostname 'smithi060' 2024-05-09T05:00:20.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40ae02c740 1 --2- 172.21.15.60:0/212887998 >> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] conn(0x562ce5fd7110 0x562ce5fb5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40ae02c740 4 client.4392 mds_command: new command op to 4261 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T05:00:20.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.114+0000 7f40ae02c740 1 -- 172.21.15.60:0/212887998 --> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ce5e78760 con 0x562ce5fd7110 2024-05-09T05:00:20.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.115+0000 7f40a8fd3640 1 --2- 172.21.15.60:0/212887998 >> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] conn(0x562ce5fd7110 0x562ce5fb5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.115+0000 7f40837fe640 10 client.4392 ms_handle_connect on v2:172.21.15.60:6834/2135423314 2024-05-09T05:00:20.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.115+0000 7f40a8fd3640 1 --2- 172.21.15.60:0/212887998 >> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] conn(0x562ce5fd7110 0x562ce5fb5690 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f1dd6a3a3770d83 server_cookie=b7872cfeb91c4828 in_seq=0 out_seq=0 2024-05-09T05:00:20.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.116+0000 7f40837fe640 1 -- 172.21.15.60:0/212887998 <== mds.0 v2:172.21.15.60:6834/2135423314 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562ce5e78760 con 0x562ce5fd7110 2024-05-09T05:00:20.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.116+0000 7f40837fe640 10 client.4392 handle_command_reply: tid=0 2024-05-09T05:00:20.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.116+0000 7f40ae02c740 1 client.4392 shutdown 2024-05-09T05:00:20.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.116+0000 7f40ae02c740 1 -- 172.21.15.60:0/212887998 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fadfd0 msgr2=0x562ce5f8f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:20.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.116+0000 7f40ae02c740 1 --2- 172.21.15.60:0/212887998 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fadfd0 0x562ce5f8f9b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f408c001bc0 tx=0x7f408c000ed0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.116+0000 7f40ae02c740 1 -- 172.21.15.60:0/212887998 shutdown_connections 2024-05-09T05:00:20.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.116+0000 7f40ae02c740 1 --2- 172.21.15.60:0/212887998 >> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] conn(0x562ce5fd7110 0x562ce5fb5690 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.116+0000 7f40ae02c740 1 --2- 172.21.15.60:0/212887998 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fadfd0 0x562ce5f8f9b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.116+0000 7f40ae02c740 1 --2- 172.21.15.60:0/212887998 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb83d0 0x562ce5f8f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.116+0000 7f40ae02c740 1 --2- 172.21.15.60:0/212887998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fb7ff0 0x562ce5faee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.116+0000 7f40837fe640 0 client.4392 ms_handle_reset on v2:172.21.15.60:6834/2135423314 2024-05-09T05:00:20.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.116+0000 7f40ae02c740 1 -- 172.21.15.60:0/212887998 >> 172.21.15.60:0/212887998 conn(0x562ce5f8ac50 msgr2=0x562ce5fb72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:20.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.116+0000 7f40ae02c740 1 -- 172.21.15.60:0/212887998 shutdown_connections 2024-05-09T05:00:20.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.116+0000 7f40ae02c740 1 -- 172.21.15.60:0/212887998 wait complete. 2024-05-09T05:00:20.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.116+0000 7f40ae02c740 20 client.4392 trim_cache size 0 max 16384 2024-05-09T05:00:20.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.135+0000 7f40ae02c740 1 Processor -- start 2024-05-09T05:00:20.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.135+0000 7f40ae02c740 1 -- start start 2024-05-09T05:00:20.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.135+0000 7f40ae02c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fadfd0 0x562ce5ff9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.135+0000 7f40ae02c740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb7ff0 0x562ce5fdd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.135+0000 7f40ae02c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fb83d0 0x562ce5fb2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.135+0000 7f40ae02c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f40a4201e50 con 0x562ce5fadfd0 2024-05-09T05:00:20.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.135+0000 7f40ae02c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f40a42011b0 con 0x562ce5fb83d0 2024-05-09T05:00:20.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.135+0000 7f40ae02c740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f40a4201800 con 0x562ce5fb7ff0 2024-05-09T05:00:20.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.136+0000 7f40a8fd3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fadfd0 0x562ce5ff9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.136+0000 7f40a8fd3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fadfd0 0x562ce5ff9750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35684/0 (socket says 172.21.15.60:35684) 2024-05-09T05:00:20.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.136+0000 7f40a8fd3640 1 -- 172.21.15.60:0/3215591349 learned_addr learned my addr 172.21.15.60:0/3215591349 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:20.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.136+0000 7f40a97d4640 1 --2- 172.21.15.60:0/3215591349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fb83d0 0x562ce5fb2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.136+0000 7f40a8fd3640 1 -- 172.21.15.60:0/3215591349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb7ff0 msgr2=0x562ce5fdd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:20.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.136+0000 7f40a3fff640 1 --2- 172.21.15.60:0/3215591349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb7ff0 0x562ce5fdd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.136+0000 7f40a8fd3640 1 --2- 172.21.15.60:0/3215591349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb7ff0 0x562ce5fdd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.136+0000 7f40a8fd3640 1 -- 172.21.15.60:0/3215591349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fb83d0 msgr2=0x562ce5fb2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:20.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.136+0000 7f40a8fd3640 1 --2- 172.21.15.60:0/3215591349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fb83d0 0x562ce5fb2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.136+0000 7f40a8fd3640 1 -- 172.21.15.60:0/3215591349 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ce5f8c5b0 con 0x562ce5fadfd0 2024-05-09T05:00:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.136+0000 7f40a97d4640 1 --2- 172.21.15.60:0/3215591349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fb83d0 0x562ce5fb2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.136+0000 7f40a8fd3640 1 --2- 172.21.15.60:0/3215591349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fadfd0 0x562ce5ff9750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f409c007fd0 tx=0x7f409c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce7e777067d39e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.136+0000 7f4099ffb640 1 -- 172.21.15.60:0/3215591349 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c027150 con 0x562ce5fadfd0 2024-05-09T05:00:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.136+0000 7f4099ffb640 1 -- 172.21.15.60:0/3215591349 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f409c0272f0 con 0x562ce5fadfd0 2024-05-09T05:00:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.136+0000 7f4099ffb640 1 -- 172.21.15.60:0/3215591349 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c0276a0 con 0x562ce5fadfd0 2024-05-09T05:00:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.137+0000 7f40ae02c740 1 -- 172.21.15.60:0/3215591349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fadfd0 msgr2=0x562ce5ff9750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.137+0000 7f40ae02c740 1 --2- 172.21.15.60:0/3215591349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fadfd0 0x562ce5ff9750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f409c007fd0 tx=0x7f409c011000 comp rx=0 tx=0).stop 2024-05-09T05:00:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.137+0000 7f40ae02c740 1 -- 172.21.15.60:0/3215591349 shutdown_connections 2024-05-09T05:00:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.137+0000 7f40ae02c740 1 --2- 172.21.15.60:0/3215591349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fb83d0 0x562ce5fb2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.137+0000 7f40ae02c740 1 --2- 172.21.15.60:0/3215591349 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb7ff0 0x562ce5fdd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.137+0000 7f40ae02c740 1 --2- 172.21.15.60:0/3215591349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fadfd0 0x562ce5ff9750 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.137+0000 7f40ae02c740 1 -- 172.21.15.60:0/3215591349 >> 172.21.15.60:0/3215591349 conn(0x562ce5f8ac50 msgr2=0x562ce5faeb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.137+0000 7f40ae02c740 1 -- 172.21.15.60:0/3215591349 shutdown_connections 2024-05-09T05:00:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.137+0000 7f40ae02c740 1 -- 172.21.15.60:0/3215591349 wait complete. 2024-05-09T05:00:20.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.137+0000 7f40ae02c740 1 -- start start 2024-05-09T05:00:20.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.138+0000 7f40ae02c740 1 Processor -- start 2024-05-09T05:00:20.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.138+0000 7f40ae02c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fadfd0 0x562ce5fb2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.138+0000 7f40ae02c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fb7ff0 0x562ce5fb3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.138+0000 7f40ae02c740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb83d0 0x562ce5fe7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.138+0000 7f40ae02c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x562ce5fd6f90 con 0x562ce5fb7ff0 2024-05-09T05:00:20.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.138+0000 7f40ae02c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562ce5fb4860 con 0x562ce5fadfd0 2024-05-09T05:00:20.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.138+0000 7f40ae02c740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x562ce5fd75e0 con 0x562ce5fb83d0 2024-05-09T05:00:20.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.138+0000 7f40ae02c740 10 client.0 _ll_register_callbacks cb 0x7f408c016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:00:20.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40a3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fb7ff0 0x562ce5fb3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40a3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fb7ff0 0x562ce5fb3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35698/0 (socket says 172.21.15.60:35698) 2024-05-09T05:00:20.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40a3fff640 1 -- 172.21.15.60:0/4240478566 learned_addr learned my addr 172.21.15.60:0/4240478566 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:20.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40a8fd3640 1 --2- 172.21.15.60:0/4240478566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fadfd0 0x562ce5fb2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40a97d4640 1 --2- 172.21.15.60:0/4240478566 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb83d0 0x562ce5fe7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40a3fff640 1 -- 172.21.15.60:0/4240478566 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb83d0 msgr2=0x562ce5fe7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:20.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40a3fff640 1 --2- 172.21.15.60:0/4240478566 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb83d0 0x562ce5fe7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40a3fff640 1 -- 172.21.15.60:0/4240478566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fadfd0 msgr2=0x562ce5fb2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:20.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40a3fff640 1 --2- 172.21.15.60:0/4240478566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fadfd0 0x562ce5fb2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40a3fff640 1 -- 172.21.15.60:0/4240478566 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ce5f8c350 con 0x562ce5fb7ff0 2024-05-09T05:00:20.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40a3fff640 1 --2- 172.21.15.60:0/4240478566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fb7ff0 0x562ce5fb3070 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f40940820b0 tx=0x7f40940863a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c575e4966813c78 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:20.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40837fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-05-09T05:00:20.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40837fe640 1 -- 172.21.15.60:0/4240478566 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40940b5060 con 0x562ce5fb7ff0 2024-05-09T05:00:20.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40837fe640 1 -- 172.21.15.60:0/4240478566 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f409408aa60 con 0x562ce5fb7ff0 2024-05-09T05:00:20.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40ae02c740 1 -- 172.21.15.60:0/4240478566 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f409c0272f0 con 0x562ce5fb7ff0 2024-05-09T05:00:20.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40837fe640 1 -- 172.21.15.60:0/4240478566 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f4094082820 con 0x562ce5fb7ff0 2024-05-09T05:00:20.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40ae02c740 10 client.4398 fetch_fsmap learned FSMap version 7 2024-05-09T05:00:20.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40ae02c740 1 -- 172.21.15.60:0/4240478566 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x562ce5fcbb30 con 0x562ce5fb7ff0 2024-05-09T05:00:20.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40837fe640 1 -- 172.21.15.60:0/4240478566 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f409c0272f0 con 0x562ce5fb7ff0 2024-05-09T05:00:20.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.139+0000 7f40837fe640 1 -- 172.21.15.60:0/4240478566 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7f4094081380 con 0x562ce5fb7ff0 2024-05-09T05:00:20.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40ae02c740 10 client.4398 fetch_fsmap finished waiting for FSMap version 7 2024-05-09T05:00:20.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40ae02c740 10 client.4398 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T05:00:20.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40ae02c740 20 client.4398 populate_metadata read hostname 'smithi060' 2024-05-09T05:00:20.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40ae02c740 1 --2- 172.21.15.60:0/4240478566 >> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] conn(0x562ce5fe7920 0x562ce5fb5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40ae02c740 4 client.4398 mds_command: new command op to 4261 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T05:00:20.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40ae02c740 1 -- 172.21.15.60:0/4240478566 --> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] -- command(tid 0: {"prefix": "status"}) -- 0x562ce5f024e0 con 0x562ce5fe7920 2024-05-09T05:00:20.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40a8fd3640 1 --2- 172.21.15.60:0/4240478566 >> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] conn(0x562ce5fe7920 0x562ce5fb5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40837fe640 10 client.4398 ms_handle_connect on v2:172.21.15.60:6834/2135423314 2024-05-09T05:00:20.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40a8fd3640 1 --2- 172.21.15.60:0/4240478566 >> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] conn(0x562ce5fe7920 0x562ce5fb5d00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7540cae80ccc5c6 server_cookie=7ba4e861a0971967 in_seq=0 out_seq=0 2024-05-09T05:00:20.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40837fe640 1 -- 172.21.15.60:0/4240478566 <== mds.0 v2:172.21.15.60:6834/2135423314 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x562ce5f024e0 con 0x562ce5fe7920 2024-05-09T05:00:20.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40837fe640 10 client.4398 handle_command_reply: tid=0 2024-05-09T05:00:20.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40ae02c740 1 client.4398 shutdown 2024-05-09T05:00:20.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40ae02c740 1 -- 172.21.15.60:0/4240478566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fb7ff0 msgr2=0x562ce5fb3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:20.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40ae02c740 1 --2- 172.21.15.60:0/4240478566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fb7ff0 0x562ce5fb3070 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f40940820b0 tx=0x7f40940863a0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40ae02c740 1 -- 172.21.15.60:0/4240478566 shutdown_connections 2024-05-09T05:00:20.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40ae02c740 1 --2- 172.21.15.60:0/4240478566 >> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] conn(0x562ce5fe7920 0x562ce5fb5d00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40ae02c740 1 --2- 172.21.15.60:0/4240478566 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562ce5fb83d0 0x562ce5fe7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40ae02c740 1 --2- 172.21.15.60:0/4240478566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562ce5fb7ff0 0x562ce5fb3070 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40837fe640 0 client.4398 ms_handle_reset on v2:172.21.15.60:6834/2135423314 2024-05-09T05:00:20.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40ae02c740 1 --2- 172.21.15.60:0/4240478566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562ce5fadfd0 0x562ce5fb2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.140+0000 7f40ae02c740 1 -- 172.21.15.60:0/4240478566 >> 172.21.15.60:0/4240478566 conn(0x562ce5f8ac50 msgr2=0x562ce5fd7c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:20.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.141+0000 7f40ae02c740 1 -- 172.21.15.60:0/4240478566 shutdown_connections 2024-05-09T05:00:20.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.141+0000 7f40ae02c740 1 -- 172.21.15.60:0/4240478566 wait complete. 2024-05-09T05:00:20.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.141+0000 7f40ae02c740 20 client.4398 trim_cache size 0 max 16384 2024-05-09T05:00:20.148 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:00:20.148 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2", 2024-05-09T05:00:20.148 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2024-05-09T05:00:20.148 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4261, 2024-05-09T05:00:20.148 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2024-05-09T05:00:20.148 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2024-05-09T05:00:20.148 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2024-05-09T05:00:20.148 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 7.3153083580000002, 2024-05-09T05:00:20.148 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 7, 2024-05-09T05:00:20.148 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 24, 2024-05-09T05:00:20.148 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 24, 2024-05-09T05:00:20.149 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 13.062909400000001 2024-05-09T05:00:20.149 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:00:20.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.141+0000 7f40ab237640 1 -- 172.21.15.60:0/120006917 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f40840762d0 msgr2=0x7f4084078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:20.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.141+0000 7f40ab237640 1 --2- 172.21.15.60:0/120006917 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f40840762d0 0x7f4084078790 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f409c000f60 tx=0x7f409c009040 comp rx=0 tx=0).stop 2024-05-09T05:00:20.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.141+0000 7f40ab237640 1 -- 172.21.15.60:0/120006917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40a41598f0 msgr2=0x7f40a415f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:20.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.141+0000 7f40ab237640 1 --2- 172.21.15.60:0/120006917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40a41598f0 0x7f40a415f480 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4094009e80 tx=0x7f4094004a40 comp rx=0 tx=0).stop 2024-05-09T05:00:20.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.141+0000 7f40ab237640 1 -- 172.21.15.60:0/120006917 shutdown_connections 2024-05-09T05:00:20.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.141+0000 7f40ab237640 1 --2- 172.21.15.60:0/120006917 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f40840762d0 0x7f4084078790 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.141+0000 7f40ab237640 1 --2- 172.21.15.60:0/120006917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40a41605d0 0x7f40a415f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.141+0000 7f40ab237640 1 --2- 172.21.15.60:0/120006917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40a41598f0 0x7f40a415f480 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.141+0000 7f40ab237640 1 --2- 172.21.15.60:0/120006917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40a4072130 0x7f40a41f4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.141+0000 7f40ab237640 1 -- 172.21.15.60:0/120006917 >> 172.21.15.60:0/120006917 conn(0x7f40a40b4850 msgr2=0x7f40a4157650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:20.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.141+0000 7f40ab237640 1 -- 172.21.15.60:0/120006917 shutdown_connections 2024-05-09T05:00:20.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.141+0000 7f40ab237640 1 -- 172.21.15.60:0/120006917 wait complete. 2024-05-09T05:00:20.153 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T05:00:20.153 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-09T05:00:20.153 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T05:00:20.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.255+0000 7f7f73a83640 1 Processor -- start 2024-05-09T05:00:20.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.255+0000 7f7f73a83640 1 -- start start 2024-05-09T05:00:20.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.256+0000 7f7f73a83640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7f6c15f500 0x7f7f6c099b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.256+0000 7f7f73a83640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f6c09a4e0 con 0x7f7f6c06fe20 2024-05-09T05:00:20.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.256+0000 7f7f73a83640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f6c072ad0 con 0x7f7f6c09a040 2024-05-09T05:00:20.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.256+0000 7f7f73a83640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f7f6c072db0 con 0x7f7f6c15f500 2024-05-09T05:00:20.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.256+0000 7f7f7181f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7f6c15f500 0x7f7f6c099b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.256+0000 7f7f72020640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7f6c06fe20 0x7f7f6c0735b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44822/0 (socket says 172.21.15.60:44822) 2024-05-09T05:00:20.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.256+0000 7f7f72020640 1 -- 172.21.15.60:0/3576967463 learned_addr learned my addr 172.21.15.60:0/3576967463 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:20.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.256+0000 7f7f7181f640 1 --2- 172.21.15.60:0/3576967463 >> v2:172.21.15.60:3300/0 conn(0x7f7f6c15f500 0x7f7f6c099b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35706/0 (socket says 172.21.15.60:35706) 2024-05-09T05:00:20.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.257+0000 7f7f7181f640 1 -- 172.21.15.60:0/3576967463 >> v1:172.21.15.114:6789/0 conn(0x7f7f6c09a040 legacy=0x7f7f6c15f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:20.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.257+0000 7f7f7181f640 1 -- 172.21.15.60:0/3576967463 >> v1:172.21.15.60:6789/0 conn(0x7f7f6c06fe20 legacy=0x7f7f6c0735b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:20.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.257+0000 7f7f7081d640 1 -- 172.21.15.60:0/3576967463 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3189091790 0 0) 0x7f7f6c09a4e0 con 0x7f7f6c06fe20 2024-05-09T05:00:20.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.257+0000 7f7f7181f640 1 -- 172.21.15.60:0/3576967463 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f6c15fff0 con 0x7f7f6c15f500 2024-05-09T05:00:20.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.257+0000 7f7f7181f640 1 --2- 172.21.15.60:0/3576967463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f6c15f500 0x7f7f6c099b00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c009d90 tx=0x7f7f5c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab31290ca439ffe7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:20.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.257+0000 7f7f7081d640 1 -- 172.21.15.60:0/3576967463 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c007740 con 0x7f7f6c15f500 2024-05-09T05:00:20.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.258+0000 7f7f7081d640 1 -- 172.21.15.60:0/3576967463 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f5c0078e0 con 0x7f7f6c15f500 2024-05-09T05:00:20.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.258+0000 7f7f7081d640 1 -- 172.21.15.60:0/3576967463 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c007cb0 con 0x7f7f6c15f500 2024-05-09T05:00:20.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.258+0000 7f7f73a83640 1 -- 172.21.15.60:0/3576967463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f6c15f500 msgr2=0x7f7f6c099b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:20.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.258+0000 7f7f73a83640 1 --2- 172.21.15.60:0/3576967463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f6c15f500 0x7f7f6c099b00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c009d90 tx=0x7f7f5c02a660 comp rx=0 tx=0).stop 2024-05-09T05:00:20.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.258+0000 7f7f73a83640 1 -- 172.21.15.60:0/3576967463 shutdown_connections 2024-05-09T05:00:20.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.258+0000 7f7f73a83640 1 --2- 172.21.15.60:0/3576967463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f6c15f500 0x7f7f6c099b00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.258+0000 7f7f73a83640 1 -- 172.21.15.60:0/3576967463 >> 172.21.15.60:0/3576967463 conn(0x7f7f6c0b4ec0 msgr2=0x7f7f6c06e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:20.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.258+0000 7f7f73a83640 1 -- 172.21.15.60:0/3576967463 shutdown_connections 2024-05-09T05:00:20.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.258+0000 7f7f73a83640 1 -- 172.21.15.60:0/3576967463 wait complete. 2024-05-09T05:00:20.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.259+0000 7f7f73a83640 1 Processor -- start 2024-05-09T05:00:20.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.259+0000 7f7f73a83640 1 -- start start 2024-05-09T05:00:20.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.260+0000 7f7f73a83640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f6c06fe20 0x7f7f6c15c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.260+0000 7f7f73a83640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f6c09a040 0x7f7f6c162160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.260+0000 7f7f73a83640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f6c15f500 0x7f7f6c1626a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.260+0000 7f7f73a83640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6c09c140 con 0x7f7f6c09a040 2024-05-09T05:00:20.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.260+0000 7f7f73a83640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6c09bfc0 con 0x7f7f6c06fe20 2024-05-09T05:00:20.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.260+0000 7f7f73a83640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7f6c073cc0 con 0x7f7f6c15f500 2024-05-09T05:00:20.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.260+0000 7f7f7101e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f6c09a040 0x7f7f6c162160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.260+0000 7f7f7101e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f6c09a040 0x7f7f6c162160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35710/0 (socket says 172.21.15.60:35710) 2024-05-09T05:00:20.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.260+0000 7f7f7101e640 1 -- 172.21.15.60:0/1729180403 learned_addr learned my addr 172.21.15.60:0/1729180403 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:20.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.260+0000 7f7f7181f640 1 --2- 172.21.15.60:0/1729180403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f6c06fe20 0x7f7f6c15c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.260+0000 7f7f72020640 1 --2- 172.21.15.60:0/1729180403 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f6c15f500 0x7f7f6c1626a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.260+0000 7f7f7101e640 1 -- 172.21.15.60:0/1729180403 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f6c15f500 msgr2=0x7f7f6c1626a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:20.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.260+0000 7f7f7101e640 1 --2- 172.21.15.60:0/1729180403 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f6c15f500 0x7f7f6c1626a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.260+0000 7f7f7101e640 1 -- 172.21.15.60:0/1729180403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f6c06fe20 msgr2=0x7f7f6c15c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:20.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.260+0000 7f7f7101e640 1 --2- 172.21.15.60:0/1729180403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f6c06fe20 0x7f7f6c15c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.260+0000 7f7f7101e640 1 -- 172.21.15.60:0/1729180403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f6c0d3d40 con 0x7f7f6c09a040 2024-05-09T05:00:20.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.260+0000 7f7f72020640 1 --2- 172.21.15.60:0/1729180403 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f6c15f500 0x7f7f6c1626a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:20.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.261+0000 7f7f7101e640 1 --2- 172.21.15.60:0/1729180403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f6c09a040 0x7f7f6c162160 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7f6000c500 tx=0x7f7f6000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:20.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.261+0000 7f7f6a7fc640 1 -- 172.21.15.60:0/1729180403 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f600052d0 con 0x7f7f6c09a040 2024-05-09T05:00:20.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.261+0000 7f7f6a7fc640 1 -- 172.21.15.60:0/1729180403 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f60005470 con 0x7f7f6c09a040 2024-05-09T05:00:20.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.261+0000 7f7f6a7fc640 1 -- 172.21.15.60:0/1729180403 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f60012870 con 0x7f7f6c09a040 2024-05-09T05:00:20.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.261+0000 7f7f73a83640 1 -- 172.21.15.60:0/1729180403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f6c0d4020 con 0x7f7f6c09a040 2024-05-09T05:00:20.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.261+0000 7f7f73a83640 1 -- 172.21.15.60:0/1729180403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f6c0d45e0 con 0x7f7f6c09a040 2024-05-09T05:00:20.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.262+0000 7f7f73a83640 1 -- 172.21.15.60:0/1729180403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f6c051040 con 0x7f7f6c09a040 2024-05-09T05:00:20.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.262+0000 7f7f6a7fc640 1 -- 172.21.15.60:0/1729180403 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7f60012a10 con 0x7f7f6c09a040 2024-05-09T05:00:20.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.263+0000 7f7f6a7fc640 1 --2- 172.21.15.60:0/1729180403 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7f580762d0 0x7f7f58078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:20.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.263+0000 7f7f6a7fc640 1 -- 172.21.15.60:0/1729180403 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7f60097570 con 0x7f7f6c09a040 2024-05-09T05:00:20.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.263+0000 7f7f7181f640 1 --2- 172.21.15.60:0/1729180403 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7f580762d0 0x7f7f58078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:20.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.264+0000 7f7f7181f640 1 --2- 172.21.15.60:0/1729180403 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7f580762d0 0x7f7f58078790 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c034000 tx=0x7f7f5c02f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:20.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.267+0000 7f7f6a7fc640 1 -- 172.21.15.60:0/1729180403 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7f60060ba0 con 0x7f7f6c09a040 2024-05-09T05:00:20.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.503+0000 7f7f73a83640 1 -- 172.21.15.60:0/1729180403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7f6c06f6a0 con 0x7f7f6c09a040 2024-05-09T05:00:20.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.503+0000 7f7f6a7fc640 1 -- 172.21.15.60:0/1729180403 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+88 (secure 0 0 0) 0x7f7f600604a0 con 0x7f7f6c09a040 2024-05-09T05:00:20.504 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-09T05:00:20.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.505+0000 7f7f73a83640 1 -- 172.21.15.60:0/1729180403 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7f580762d0 msgr2=0x7f7f58078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:20.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.506+0000 7f7f73a83640 1 --2- 172.21.15.60:0/1729180403 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7f580762d0 0x7f7f58078790 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c034000 tx=0x7f7f5c02f000 comp rx=0 tx=0).stop 2024-05-09T05:00:20.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.506+0000 7f7f73a83640 1 -- 172.21.15.60:0/1729180403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f6c09a040 msgr2=0x7f7f6c162160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.506+0000 7f7f73a83640 1 --2- 172.21.15.60:0/1729180403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f6c09a040 0x7f7f6c162160 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7f6000c500 tx=0x7f7f6000c9c0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.506+0000 7f7f73a83640 1 -- 172.21.15.60:0/1729180403 shutdown_connections 2024-05-09T05:00:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.506+0000 7f7f73a83640 1 --2- 172.21.15.60:0/1729180403 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7f580762d0 0x7f7f58078790 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.506+0000 7f7f73a83640 1 --2- 172.21.15.60:0/1729180403 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7f6c15f500 0x7f7f6c1626a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.506+0000 7f7f73a83640 1 --2- 172.21.15.60:0/1729180403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f6c09a040 0x7f7f6c162160 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.506+0000 7f7f73a83640 1 --2- 172.21.15.60:0/1729180403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f6c06fe20 0x7f7f6c15c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:20.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.506+0000 7f7f73a83640 1 -- 172.21.15.60:0/1729180403 >> 172.21.15.60:0/1729180403 conn(0x7f7f6c0b4ec0 msgr2=0x7f7f6c1602b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:20.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.506+0000 7f7f73a83640 1 -- 172.21.15.60:0/1729180403 shutdown_connections 2024-05-09T05:00:20.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20.506+0000 7f7f73a83640 1 -- 172.21.15.60:0/1729180403 wait complete. 2024-05-09T05:00:20.517 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-09T05:00:20.517 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T05:00:20.517 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2024-05-09T05:00:20.517 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-05-09T05:00:20.517 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-09T05:00:20.517 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-09T05:00:20.517 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T05:00:20.517 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T05:00:20.517 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-05-09T05:00:20.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:20.518 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2024-05-09T05:00:20.534 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T05:00:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T05:00:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T05:00:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2024-05-09T05:00:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:20.534 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:00:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:20.534 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2024-05-09T05:00:20.535 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:00:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2024-05-09T05:00:20.535 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T05:00:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2024-05-09T05:00:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T05:00:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 42744sec preferred_lft 42744sec 2024-05-09T05:00:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2024-05-09T05:00:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:20.535 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:00:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:20.535 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2024-05-09T05:00:20.536 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-09T05:00:20.536 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:20.536 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-05-09T05:00:20.536 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link add name ceph-brx type bridge 2024-05-09T05:00:20.536 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip addr flush dev ceph-brx 2024-05-09T05:00:20.536 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set ceph-brx up 2024-05-09T05:00:20.536 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-09T05:00:20.536 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-05-09T05:00:20.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:20.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:20.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:20.758 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T05:00:20.784 INFO:teuthology.orchestra.run.smithi060.stdout:1 2024-05-09T05:00:20.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:20.786 DEBUG:teuthology.orchestra.run.smithi060:> route 2024-05-09T05:00:20.850 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2024-05-09T05:00:20.851 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T05:00:20.851 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T05:00:20.851 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T05:00:20.851 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T05:00:20.851 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:20.851 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-05-09T05:00:20.852 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T05:00:20.852 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T05:00:20.852 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T05:00:20.852 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-05-09T05:00:20.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:21.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:21.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:21.195 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2024-05-09T05:00:21.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:21.211 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list-id 2024-05-09T05:00:21.267 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:21.267 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-05-09T05:00:21.267 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T05:00:21.267 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-05-09T05:00:21.267 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-05-09T05:00:21.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:21.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:21.387 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-05-09T05:00:21.388 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:21.388 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-05-09T05:00:21.388 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-05-09T05:00:21.388 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-09T05:00:21.388 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-05-09T05:00:21.388 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-05-09T05:00:21.388 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-05-09T05:00:21.388 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-05-09T05:00:21.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:21.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:21.518 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:21.518 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-05-09T05:00:21.518 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2024-05-09T05:00:21.518 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set dev brx.0 master ceph-brx 2024-05-09T05:00:21.518 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-05-09T05:00:21.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:21.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:21.597 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-05-09T05:00:21.597 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T05:00:21.598 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-05-09T05:00:21.614 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-05-09T05:00:21.614 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T05:00:21.614 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-05-09T05:00:21.670 DEBUG:teuthology.orchestra.run.smithi060:> sudo modprobe fuse 2024-05-09T05:00:21.736 DEBUG:teuthology.orchestra.run.smithi060:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T05:00:21.794 INFO:teuthology.orchestra.run.smithi060.stdout:/proc 2024-05-09T05:00:21.794 INFO:teuthology.orchestra.run.smithi060.stdout:/sys 2024-05-09T05:00:21.794 INFO:teuthology.orchestra.run.smithi060.stdout:/dev 2024-05-09T05:00:21.794 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/security 2024-05-09T05:00:21.794 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/shm 2024-05-09T05:00:21.795 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/pts 2024-05-09T05:00:21.795 INFO:teuthology.orchestra.run.smithi060.stdout:/run 2024-05-09T05:00:21.795 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/cgroup 2024-05-09T05:00:21.795 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/pstore 2024-05-09T05:00:21.795 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/bpf 2024-05-09T05:00:21.795 INFO:teuthology.orchestra.run.smithi060.stdout:/ 2024-05-09T05:00:21.795 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/selinux 2024-05-09T05:00:21.795 INFO:teuthology.orchestra.run.smithi060.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T05:00:21.795 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/mqueue 2024-05-09T05:00:21.795 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/hugepages 2024-05-09T05:00:21.795 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/debug 2024-05-09T05:00:21.796 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/tracing 2024-05-09T05:00:21.796 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/fuse/connections 2024-05-09T05:00:21.796 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/config 2024-05-09T05:00:21.796 INFO:teuthology.orchestra.run.smithi060.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T05:00:21.796 INFO:teuthology.orchestra.run.smithi060.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T05:00:21.796 INFO:teuthology.orchestra.run.smithi060.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T05:00:21.796 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T05:00:21.796 INFO:teuthology.orchestra.run.smithi060.stdout:/run/user/1000 2024-05-09T05:00:21.796 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/ceph 2024-05-09T05:00:21.796 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T05:00:21.796 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T05:00:21.796 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T05:00:21.796 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T05:00:21.796 INFO:teuthology.orchestra.run.smithi060.stdout:/run/netns 2024-05-09T05:00:21.797 INFO:teuthology.orchestra.run.smithi060.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T05:00:21.797 INFO:teuthology.orchestra.run.smithi060.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T05:00:21.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:21.797 DEBUG:teuthology.orchestra.run.smithi060:> ls /sys/fs/fuse/connections 2024-05-09T05:00:21.850 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T05:00:21.850 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-05-09T05:00:21.893 DEBUG:teuthology.orchestra.run.smithi060:> sudo modprobe fuse 2024-05-09T05:00:21.918 DEBUG:teuthology.orchestra.run.smithi060:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T05:00:21.975 INFO:teuthology.orchestra.run.smithi060.stdout:/proc 2024-05-09T05:00:21.975 INFO:teuthology.orchestra.run.smithi060.stdout:/sys 2024-05-09T05:00:21.975 INFO:teuthology.orchestra.run.smithi060.stdout:/dev 2024-05-09T05:00:21.975 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/security 2024-05-09T05:00:21.976 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/shm 2024-05-09T05:00:21.976 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/pts 2024-05-09T05:00:21.976 INFO:teuthology.orchestra.run.smithi060.stdout:/run 2024-05-09T05:00:21.976 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/cgroup 2024-05-09T05:00:21.976 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/pstore 2024-05-09T05:00:21.976 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/bpf 2024-05-09T05:00:21.976 INFO:teuthology.orchestra.run.smithi060.stdout:/ 2024-05-09T05:00:21.976 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/selinux 2024-05-09T05:00:21.976 INFO:teuthology.orchestra.run.smithi060.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T05:00:21.976 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/mqueue 2024-05-09T05:00:21.976 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/hugepages 2024-05-09T05:00:21.976 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/debug 2024-05-09T05:00:21.976 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/tracing 2024-05-09T05:00:21.976 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/fuse/connections 2024-05-09T05:00:21.977 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/config 2024-05-09T05:00:21.977 INFO:teuthology.orchestra.run.smithi060.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T05:00:21.978 INFO:teuthology.orchestra.run.smithi060.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T05:00:21.978 INFO:teuthology.orchestra.run.smithi060.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T05:00:21.978 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T05:00:21.978 INFO:teuthology.orchestra.run.smithi060.stdout:/run/user/1000 2024-05-09T05:00:21.978 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/ceph 2024-05-09T05:00:21.978 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T05:00:21.978 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T05:00:21.978 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T05:00:21.979 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T05:00:21.979 INFO:teuthology.orchestra.run.smithi060.stdout:/run/netns 2024-05-09T05:00:21.979 INFO:teuthology.orchestra.run.smithi060.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T05:00:21.979 INFO:teuthology.orchestra.run.smithi060.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T05:00:21.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:21.979 DEBUG:teuthology.orchestra.run.smithi060:> ls /sys/fs/fuse/connections 2024-05-09T05:00:22.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi060.stderr:2024-05-09T05:00:22.001+0000 7faac5722480 -1 init, newargv = 0x55cde1d8c400 newargc=15 2024-05-09T05:00:22.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi060.stderr:ceph-fuse[41956]: starting ceph client 2024-05-09T05:00:22.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi060.stderr:ceph-fuse[41956]: starting fuse 2024-05-09T05:00:22.018 INFO:teuthology.orchestra.run.smithi060.stdout:41 2024-05-09T05:00:22.019 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-05-09T05:00:22.019 DEBUG:teuthology.orchestra.run.smithi060:> sudo stdin-killer -- python3 -c ' 2024-05-09T05:00:22.019 DEBUG:teuthology.orchestra.run.smithi060:> import glob 2024-05-09T05:00:22.019 DEBUG:teuthology.orchestra.run.smithi060:> import re 2024-05-09T05:00:22.019 DEBUG:teuthology.orchestra.run.smithi060:> import os 2024-05-09T05:00:22.019 DEBUG:teuthology.orchestra.run.smithi060:> import subprocess 2024-05-09T05:00:22.019 DEBUG:teuthology.orchestra.run.smithi060:> 2024-05-09T05:00:22.019 DEBUG:teuthology.orchestra.run.smithi060:> def _find_admin_socket(client_name): 2024-05-09T05:00:22.019 DEBUG:teuthology.orchestra.run.smithi060:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-05-09T05:00:22.019 DEBUG:teuthology.orchestra.run.smithi060:> files = glob.glob(asok_path) 2024-05-09T05:00:22.019 DEBUG:teuthology.orchestra.run.smithi060:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-05-09T05:00:22.020 DEBUG:teuthology.orchestra.run.smithi060:> 2024-05-09T05:00:22.020 DEBUG:teuthology.orchestra.run.smithi060:> # Given a non-glob path, it better be there 2024-05-09T05:00:22.020 DEBUG:teuthology.orchestra.run.smithi060:> if "*" not in asok_path: 2024-05-09T05:00:22.020 DEBUG:teuthology.orchestra.run.smithi060:> assert(len(files) == 1) 2024-05-09T05:00:22.020 DEBUG:teuthology.orchestra.run.smithi060:> return files[0] 2024-05-09T05:00:22.020 DEBUG:teuthology.orchestra.run.smithi060:> 2024-05-09T05:00:22.020 DEBUG:teuthology.orchestra.run.smithi060:> for f in files: 2024-05-09T05:00:22.020 DEBUG:teuthology.orchestra.run.smithi060:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T05:00:22.020 DEBUG:teuthology.orchestra.run.smithi060:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T05:00:22.020 DEBUG:teuthology.orchestra.run.smithi060:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T05:00:22.020 DEBUG:teuthology.orchestra.run.smithi060:> contents = proc_f.read() 2024-05-09T05:00:22.020 DEBUG:teuthology.orchestra.run.smithi060:> if mountpoint in contents: 2024-05-09T05:00:22.020 DEBUG:teuthology.orchestra.run.smithi060:> return f 2024-05-09T05:00:22.020 DEBUG:teuthology.orchestra.run.smithi060:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T05:00:22.020 DEBUG:teuthology.orchestra.run.smithi060:> 2024-05-09T05:00:22.020 DEBUG:teuthology.orchestra.run.smithi060:> print(_find_admin_socket("client.admin")) 2024-05-09T05:00:22.020 DEBUG:teuthology.orchestra.run.smithi060:> ' 2024-05-09T05:00:22.140 INFO:teuthology.orchestra.run.smithi060.stdout:/var/run/ceph/ceph-client.admin.41956.asok 2024-05-09T05:00:22.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:22.151 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.41956.asok 2024-05-09T05:00:22.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:22.152 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.41956.asok status 2024-05-09T05:00:22.293 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:00:22.293 INFO:teuthology.orchestra.run.smithi060.stdout: "metadata": { 2024-05-09T05:00:22.293 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_sha1": "165a15e51d08073b5317c443415b0d8a7141b1ab", 2024-05-09T05:00:22.293 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_version": "ceph version 19.0.0-3568-g165a15e5 (165a15e51d08073b5317c443415b0d8a7141b1ab) squid (dev)", 2024-05-09T05:00:22.293 INFO:teuthology.orchestra.run.smithi060.stdout: "entity_id": "admin", 2024-05-09T05:00:22.293 INFO:teuthology.orchestra.run.smithi060.stdout: "hostname": "smithi060", 2024-05-09T05:00:22.293 INFO:teuthology.orchestra.run.smithi060.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-05-09T05:00:22.294 INFO:teuthology.orchestra.run.smithi060.stdout: "pid": "41956", 2024-05-09T05:00:22.294 INFO:teuthology.orchestra.run.smithi060.stdout: "root": "/" 2024-05-09T05:00:22.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:22.294 INFO:teuthology.orchestra.run.smithi060.stdout: "dentry_count": 0, 2024-05-09T05:00:22.294 INFO:teuthology.orchestra.run.smithi060.stdout: "dentry_pinned_count": 0, 2024-05-09T05:00:22.294 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4413, 2024-05-09T05:00:22.294 INFO:teuthology.orchestra.run.smithi060.stdout: "inst": { 2024-05-09T05:00:22.294 INFO:teuthology.orchestra.run.smithi060.stdout: "name": { 2024-05-09T05:00:22.294 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "client", 2024-05-09T05:00:22.294 INFO:teuthology.orchestra.run.smithi060.stdout: "num": 4413 2024-05-09T05:00:22.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:22.294 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": { 2024-05-09T05:00:22.295 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-05-09T05:00:22.295 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "192.168.0.1:0", 2024-05-09T05:00:22.295 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 3939886114 2024-05-09T05:00:22.295 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:22.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:22.295 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": { 2024-05-09T05:00:22.295 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-05-09T05:00:22.295 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "192.168.0.1:0", 2024-05-09T05:00:22.295 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 3939886114 2024-05-09T05:00:22.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:22.295 INFO:teuthology.orchestra.run.smithi060.stdout: "inst_str": "client.4413 192.168.0.1:0/3939886114", 2024-05-09T05:00:22.295 INFO:teuthology.orchestra.run.smithi060.stdout: "addr_str": "192.168.0.1:0/3939886114", 2024-05-09T05:00:22.296 INFO:teuthology.orchestra.run.smithi060.stdout: "inode_count": 1, 2024-05-09T05:00:22.296 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_epoch": 7, 2024-05-09T05:00:22.296 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_epoch": 24, 2024-05-09T05:00:22.296 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_epoch_barrier": 0, 2024-05-09T05:00:22.296 INFO:teuthology.orchestra.run.smithi060.stdout: "blocklisted": false, 2024-05-09T05:00:22.296 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs" 2024-05-09T05:00:22.296 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:00:22.300 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:00:22.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:22.302 DEBUG:teuthology.orchestra.run.smithi060:> stat --file-system '--printf=%T 2024-05-09T05:00:22.302 DEBUG:teuthology.orchestra.run.smithi060:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-05-09T05:00:22.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:00:22.361 INFO:teuthology.orchestra.run.smithi060.stdout:fuseblk 2024-05-09T05:00:22.361 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-05-09T05:00:22.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:22.361 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-05-09T05:00:22.465 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-05-09T05:00:22.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T05:00:22.576 INFO:teuthology.orchestra.run.smithi060.stdout:# file: . 2024-05-09T05:00:22.576 INFO:teuthology.orchestra.run.smithi060.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-05-09T05:00:22.576 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:22.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:22.582 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:00:22.582 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:00:22.599 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T05:00:22.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.745+0000 7f38fd1c9640 1 Processor -- start 2024-05-09T05:00:22.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.745+0000 7f38fd1c9640 1 -- start start 2024-05-09T05:00:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.746+0000 7f38fd1c9640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f38f815f300 0x7f38f81639f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.746+0000 7f38fd1c9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38f806ff60 con 0x7f38f806f9f0 2024-05-09T05:00:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.746+0000 7f38fd1c9640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38f8070240 con 0x7f38f81606f0 2024-05-09T05:00:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.746+0000 7f38fd1c9640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f38f8070520 con 0x7f38f815f300 2024-05-09T05:00:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.746+0000 7f38f659c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f38f806f9f0 0x7f38f80d33f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44832/0 (socket says 172.21.15.60:44832) 2024-05-09T05:00:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.746+0000 7f38f659c640 1 -- 172.21.15.60:0/3611013869 learned_addr learned my addr 172.21.15.60:0/3611013869 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.746+0000 7f38f6d9d640 1 --2- 172.21.15.60:0/3611013869 >> v2:172.21.15.114:3300/0 conn(0x7f38f815f300 0x7f38f81639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.746+0000 7f38f5d9b640 1 -- 172.21.15.60:0/3611013869 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 71828292 0 0) 0x7f38f806ff60 con 0x7f38f806f9f0 2024-05-09T05:00:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.746+0000 7f38f5d9b640 1 -- 172.21.15.60:0/3611013869 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38e4003520 con 0x7f38f806f9f0 2024-05-09T05:00:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.747+0000 7f38f5d9b640 1 -- 172.21.15.60:0/3611013869 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 543305046 0 0) 0x7f38e4003520 con 0x7f38f806f9f0 2024-05-09T05:00:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.747+0000 7f38f6d9d640 1 -- 172.21.15.60:0/3611013869 >> v1:172.21.15.114:6789/0 conn(0x7f38f81606f0 legacy=0x7f38f806ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.747+0000 7f38f6d9d640 1 -- 172.21.15.60:0/3611013869 >> v1:172.21.15.60:6789/0 conn(0x7f38f806f9f0 legacy=0x7f38f80d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.747+0000 7f38f6d9d640 1 -- 172.21.15.60:0/3611013869 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38f806e740 con 0x7f38f815f300 2024-05-09T05:00:22.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.748+0000 7f38f6d9d640 1 --2- 172.21.15.60:0/3611013869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38f815f300 0x7f38f81639f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f38e00097a0 tx=0x7f38e002a0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22f0faf36a138a61 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:22.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.748+0000 7f38f5d9b640 1 -- 172.21.15.60:0/3611013869 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e0037070 con 0x7f38f815f300 2024-05-09T05:00:22.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.748+0000 7f38f5d9b640 1 -- 172.21.15.60:0/3611013869 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38e002ab60 con 0x7f38f815f300 2024-05-09T05:00:22.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.748+0000 7f38f5d9b640 1 -- 172.21.15.60:0/3611013869 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e80016f0 con 0x7f38f815f300 2024-05-09T05:00:22.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.748+0000 7f38fd1c9640 1 -- 172.21.15.60:0/3611013869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38f815f300 msgr2=0x7f38f81639f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:22.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.748+0000 7f38fd1c9640 1 --2- 172.21.15.60:0/3611013869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38f815f300 0x7f38f81639f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f38e00097a0 tx=0x7f38e002a0d0 comp rx=0 tx=0).stop 2024-05-09T05:00:22.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.748+0000 7f38fd1c9640 1 -- 172.21.15.60:0/3611013869 shutdown_connections 2024-05-09T05:00:22.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.748+0000 7f38fd1c9640 1 --2- 172.21.15.60:0/3611013869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38f815f300 0x7f38f81639f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:22.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.748+0000 7f38fd1c9640 1 -- 172.21.15.60:0/3611013869 >> 172.21.15.60:0/3611013869 conn(0x7f38f809b850 msgr2=0x7f38f809bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:22.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.748+0000 7f38fd1c9640 1 -- 172.21.15.60:0/3611013869 shutdown_connections 2024-05-09T05:00:22.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.748+0000 7f38fd1c9640 1 -- 172.21.15.60:0/3611013869 wait complete. 2024-05-09T05:00:22.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.750+0000 7f38fd1c9640 1 Processor -- start 2024-05-09T05:00:22.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.750+0000 7f38fd1c9640 1 -- start start 2024-05-09T05:00:22.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.751+0000 7f38fd1c9640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38f806f9f0 0x7f38f815b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:22.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.751+0000 7f38fd1c9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38f815f300 0x7f38f81f5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:22.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.751+0000 7f38fd1c9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38f81606f0 0x7f38f81f5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:22.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.751+0000 7f38fd1c9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f38f80d3b00 con 0x7f38f815f300 2024-05-09T05:00:22.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.751+0000 7f38fd1c9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f38f8163f30 con 0x7f38f81606f0 2024-05-09T05:00:22.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.751+0000 7f38fd1c9640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f38f80d3c80 con 0x7f38f806f9f0 2024-05-09T05:00:22.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.751+0000 7f38f659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38f815f300 0x7f38f81f5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:22.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.751+0000 7f38f659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38f815f300 0x7f38f81f5000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35714/0 (socket says 172.21.15.60:35714) 2024-05-09T05:00:22.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.751+0000 7f38f659c640 1 -- 172.21.15.60:0/1293641314 learned_addr learned my addr 172.21.15.60:0/1293641314 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:22.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.751+0000 7f38f759e640 1 --2- 172.21.15.60:0/1293641314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38f81606f0 0x7f38f81f5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:22.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.751+0000 7f38f6d9d640 1 --2- 172.21.15.60:0/1293641314 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38f806f9f0 0x7f38f815b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:22.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.751+0000 7f38f659c640 1 -- 172.21.15.60:0/1293641314 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38f806f9f0 msgr2=0x7f38f815b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:22.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.751+0000 7f38f659c640 1 --2- 172.21.15.60:0/1293641314 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38f806f9f0 0x7f38f815b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:22.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.751+0000 7f38f659c640 1 -- 172.21.15.60:0/1293641314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38f81606f0 msgr2=0x7f38f81f5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:22.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.751+0000 7f38f659c640 1 --2- 172.21.15.60:0/1293641314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38f81606f0 0x7f38f81f5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:22.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.751+0000 7f38f659c640 1 -- 172.21.15.60:0/1293641314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38f809f540 con 0x7f38f815f300 2024-05-09T05:00:22.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.751+0000 7f38f659c640 1 --2- 172.21.15.60:0/1293641314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38f815f300 0x7f38f81f5000 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f38e800c730 tx=0x7f38e800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:22.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.752+0000 7f38ef7fe640 1 -- 172.21.15.60:0/1293641314 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e8017070 con 0x7f38f815f300 2024-05-09T05:00:22.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.752+0000 7f38ef7fe640 1 -- 172.21.15.60:0/1293641314 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38e80072d0 con 0x7f38f815f300 2024-05-09T05:00:22.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.752+0000 7f38fd1c9640 1 -- 172.21.15.60:0/1293641314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38f81f6410 con 0x7f38f815f300 2024-05-09T05:00:22.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.752+0000 7f38ef7fe640 1 -- 172.21.15.60:0/1293641314 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e80047e0 con 0x7f38f815f300 2024-05-09T05:00:22.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.752+0000 7f38fd1c9640 1 -- 172.21.15.60:0/1293641314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38f81f6110 con 0x7f38f815f300 2024-05-09T05:00:22.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.753+0000 7f38fd1c9640 1 -- 172.21.15.60:0/1293641314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38f81f65e0 con 0x7f38f815f300 2024-05-09T05:00:22.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.753+0000 7f38ef7fe640 1 -- 172.21.15.60:0/1293641314 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f38e801a070 con 0x7f38f815f300 2024-05-09T05:00:22.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.754+0000 7f38ef7fe640 1 --2- 172.21.15.60:0/1293641314 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f38d80762d0 0x7f38d8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:22.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.754+0000 7f38ef7fe640 1 -- 172.21.15.60:0/1293641314 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f38e8096cf0 con 0x7f38f815f300 2024-05-09T05:00:22.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.754+0000 7f38f6d9d640 1 --2- 172.21.15.60:0/1293641314 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f38d80762d0 0x7f38d8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:22.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.755+0000 7f38f6d9d640 1 --2- 172.21.15.60:0/1293641314 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f38d80762d0 0x7f38d8078790 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f38e0034000 tx=0x7f38e002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:22.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.759+0000 7f38ef7fe640 1 -- 172.21.15.60:0/1293641314 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f38e809b070 con 0x7f38f815f300 2024-05-09T05:00:22.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.981+0000 7f38fd1c9640 1 -- 172.21.15.60:0/1293641314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f38f8051040 con 0x7f38f815f300 2024-05-09T05:00:22.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.981+0000 7f38ef7fe640 1 -- 172.21.15.60:0/1293641314 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f38e8068390 con 0x7f38f815f300 2024-05-09T05:00:22.982 INFO:teuthology.orchestra.run.smithi060.stderr:listed 0 entries 2024-05-09T05:00:22.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.983+0000 7f38fd1c9640 1 -- 172.21.15.60:0/1293641314 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f38d80762d0 msgr2=0x7f38d8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:22.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.983+0000 7f38fd1c9640 1 --2- 172.21.15.60:0/1293641314 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f38d80762d0 0x7f38d8078790 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f38e0034000 tx=0x7f38e002e000 comp rx=0 tx=0).stop 2024-05-09T05:00:22.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.983+0000 7f38fd1c9640 1 -- 172.21.15.60:0/1293641314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38f815f300 msgr2=0x7f38f81f5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:22.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.983+0000 7f38fd1c9640 1 --2- 172.21.15.60:0/1293641314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38f815f300 0x7f38f81f5000 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f38e800c730 tx=0x7f38e800cbf0 comp rx=0 tx=0).stop 2024-05-09T05:00:22.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.983+0000 7f38fd1c9640 1 -- 172.21.15.60:0/1293641314 shutdown_connections 2024-05-09T05:00:22.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.983+0000 7f38fd1c9640 1 --2- 172.21.15.60:0/1293641314 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f38d80762d0 0x7f38d8078790 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:22.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.983+0000 7f38fd1c9640 1 --2- 172.21.15.60:0/1293641314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f38f81606f0 0x7f38f81f5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:22.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.983+0000 7f38fd1c9640 1 --2- 172.21.15.60:0/1293641314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38f815f300 0x7f38f81f5000 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:22.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.983+0000 7f38fd1c9640 1 --2- 172.21.15.60:0/1293641314 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f38f806f9f0 0x7f38f815b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:22.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.984+0000 7f38fd1c9640 1 -- 172.21.15.60:0/1293641314 >> 172.21.15.60:0/1293641314 conn(0x7f38f809b850 msgr2=0x7f38f809a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:22.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.984+0000 7f38fd1c9640 1 -- 172.21.15.60:0/1293641314 shutdown_connections 2024-05-09T05:00:22.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:22.984+0000 7f38fd1c9640 1 -- 172.21.15.60:0/1293641314 wait complete. 2024-05-09T05:00:22.995 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:00:22.996 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:00:23.013 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T05:00:23.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.157+0000 7f93f54cc640 1 Processor -- start 2024-05-09T05:00:23.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.157+0000 7f93f54cc640 1 -- start start 2024-05-09T05:00:23.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.157+0000 7f93f54cc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f93f006fe20 0x7f93f00702d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:23.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.157+0000 7f93f54cc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93f00736d0 0x7f93f0073b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:23.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.157+0000 7f93f54cc640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93f00d08e0 con 0x7f93f015f300 2024-05-09T05:00:23.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.157+0000 7f93f54cc640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f93f0073240 con 0x7f93f006fe20 2024-05-09T05:00:23.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.157+0000 7f93f54cc640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f93f00d0bc0 con 0x7f93f00736d0 2024-05-09T05:00:23.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.158+0000 7f93ee7fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f93f006fe20 0x7f93f00702d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:23.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.158+0000 7f93eeffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f93f015f300 0x7f93f0072220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44834/0 (socket says 172.21.15.60:44834) 2024-05-09T05:00:23.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.158+0000 7f93eeffd640 1 -- 172.21.15.60:0/1046673528 learned_addr learned my addr 172.21.15.60:0/1046673528 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:23.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.158+0000 7f93ef7fe640 1 --2- 172.21.15.60:0/1046673528 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93f00736d0 0x7f93f0073b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:23.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.158+0000 7f93ee7fc640 1 -- 172.21.15.60:0/1046673528 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93f00736d0 msgr2=0x7f93f0073b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:23.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.158+0000 7f93ee7fc640 1 --2- 172.21.15.60:0/1046673528 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93f00736d0 0x7f93f0073b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:23.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.158+0000 7f93ee7fc640 1 -- 172.21.15.60:0/1046673528 >> v1:172.21.15.60:6789/0 conn(0x7f93f015f300 legacy=0x7f93f0072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:23.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.158+0000 7f93ee7fc640 1 -- 172.21.15.60:0/1046673528 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93f0072ed0 con 0x7f93f006fe20 2024-05-09T05:00:23.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.158+0000 7f93ef7fe640 1 --2- 172.21.15.60:0/1046673528 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93f00736d0 0x7f93f0073b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:23.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.159+0000 7f93ee7fc640 1 --2- 172.21.15.60:0/1046673528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93f006fe20 0x7f93f00702d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f93e40097a0 tx=0x7f93e402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47389b308dc7fd2c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:23.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.159+0000 7f93edffb640 1 -- 172.21.15.60:0/1046673528 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e4037070 con 0x7f93f006fe20 2024-05-09T05:00:23.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.159+0000 7f93edffb640 1 -- 172.21.15.60:0/1046673528 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93e4032440 con 0x7f93f006fe20 2024-05-09T05:00:23.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.159+0000 7f93edffb640 1 -- 172.21.15.60:0/1046673528 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e4032810 con 0x7f93f006fe20 2024-05-09T05:00:23.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.159+0000 7f93f54cc640 1 -- 172.21.15.60:0/1046673528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93f006fe20 msgr2=0x7f93f00702d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:23.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.159+0000 7f93f54cc640 1 --2- 172.21.15.60:0/1046673528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93f006fe20 0x7f93f00702d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f93e40097a0 tx=0x7f93e402a180 comp rx=0 tx=0).stop 2024-05-09T05:00:23.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.159+0000 7f93f54cc640 1 -- 172.21.15.60:0/1046673528 shutdown_connections 2024-05-09T05:00:23.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.159+0000 7f93f54cc640 1 --2- 172.21.15.60:0/1046673528 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93f00736d0 0x7f93f0073b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.159+0000 7f93f54cc640 1 --2- 172.21.15.60:0/1046673528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93f006fe20 0x7f93f00702d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.159+0000 7f93f54cc640 1 -- 172.21.15.60:0/1046673528 >> 172.21.15.60:0/1046673528 conn(0x7f93f00b4ef0 msgr2=0x7f93f006e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.159+0000 7f93f54cc640 1 -- 172.21.15.60:0/1046673528 shutdown_connections 2024-05-09T05:00:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.160+0000 7f93f54cc640 1 -- 172.21.15.60:0/1046673528 wait complete. 2024-05-09T05:00:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.161+0000 7f93f54cc640 1 Processor -- start 2024-05-09T05:00:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.161+0000 7f93f54cc640 1 -- start start 2024-05-09T05:00:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93f54cc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93f006fe20 0x7f93f01f9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93f54cc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93f00736d0 0x7f93f00b22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93f54cc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93f015f300 0x7f93f01613d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93f54cc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f93f00d04f0 con 0x7f93f00736d0 2024-05-09T05:00:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93f54cc640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f93f009c1e0 con 0x7f93f006fe20 2024-05-09T05:00:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93f54cc640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f93f00d0670 con 0x7f93f015f300 2024-05-09T05:00:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93ee7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93f00736d0 0x7f93f00b22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93ee7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93f00736d0 0x7f93f00b22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35740/0 (socket says 172.21.15.60:35740) 2024-05-09T05:00:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93ee7fc640 1 -- 172.21.15.60:0/1817289758 learned_addr learned my addr 172.21.15.60:0/1817289758 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93eeffd640 1 --2- 172.21.15.60:0/1817289758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93f006fe20 0x7f93f01f9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93ef7fe640 1 --2- 172.21.15.60:0/1817289758 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93f015f300 0x7f93f01613d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93ee7fc640 1 -- 172.21.15.60:0/1817289758 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93f015f300 msgr2=0x7f93f01613d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93ee7fc640 1 --2- 172.21.15.60:0/1817289758 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93f015f300 0x7f93f01613d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93ee7fc640 1 -- 172.21.15.60:0/1817289758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93f006fe20 msgr2=0x7f93f01f9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93ee7fc640 1 --2- 172.21.15.60:0/1817289758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93f006fe20 0x7f93f01f9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93ee7fc640 1 -- 172.21.15.60:0/1817289758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93f00b2d70 con 0x7f93f00736d0 2024-05-09T05:00:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93ee7fc640 1 --2- 172.21.15.60:0/1817289758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93f00736d0 0x7f93f00b22b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f93e4035040 tx=0x7f93e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93d77fe640 1 -- 172.21.15.60:0/1817289758 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e4037040 con 0x7f93f00736d0 2024-05-09T05:00:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93d77fe640 1 -- 172.21.15.60:0/1817289758 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93e4030770 con 0x7f93f00736d0 2024-05-09T05:00:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93f54cc640 1 -- 172.21.15.60:0/1817289758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93f00b3050 con 0x7f93f00736d0 2024-05-09T05:00:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93d77fe640 1 -- 172.21.15.60:0/1817289758 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e4030b80 con 0x7f93f00736d0 2024-05-09T05:00:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.162+0000 7f93f54cc640 1 -- 172.21.15.60:0/1817289758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93f01f8910 con 0x7f93f00736d0 2024-05-09T05:00:23.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.163+0000 7f93f54cc640 1 -- 172.21.15.60:0/1817289758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93f0051040 con 0x7f93f00736d0 2024-05-09T05:00:23.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.164+0000 7f93d77fe640 1 -- 172.21.15.60:0/1817289758 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f93e4044020 con 0x7f93f00736d0 2024-05-09T05:00:23.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.165+0000 7f93d77fe640 1 --2- 172.21.15.60:0/1817289758 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f93dc0762d0 0x7f93dc078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.165+0000 7f93d77fe640 1 -- 172.21.15.60:0/1817289758 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f93e40b7fd0 con 0x7f93f00736d0 2024-05-09T05:00:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.165+0000 7f93eeffd640 1 --2- 172.21.15.60:0/1817289758 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f93dc0762d0 0x7f93dc078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.165+0000 7f93eeffd640 1 --2- 172.21.15.60:0/1817289758 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f93dc0762d0 0x7f93dc078790 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f93e0000f60 tx=0x7f93e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:23.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.173+0000 7f93d77fe640 1 -- 172.21.15.60:0/1817289758 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f93e4032e20 con 0x7f93f00736d0 2024-05-09T05:00:23.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.379+0000 7f93f54cc640 1 -- 172.21.15.60:0/1817289758 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f93f00aceb0 con 0x7f93f00736d0 2024-05-09T05:00:23.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.379+0000 7f93d77fe640 1 -- 172.21.15.60:0/1817289758 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f93e4081550 con 0x7f93f00736d0 2024-05-09T05:00:23.380 INFO:teuthology.orchestra.run.smithi060.stderr:listed 0 entries 2024-05-09T05:00:23.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.381+0000 7f93f54cc640 1 -- 172.21.15.60:0/1817289758 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f93dc0762d0 msgr2=0x7f93dc078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:23.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.381+0000 7f93f54cc640 1 --2- 172.21.15.60:0/1817289758 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f93dc0762d0 0x7f93dc078790 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f93e0000f60 tx=0x7f93e0009040 comp rx=0 tx=0).stop 2024-05-09T05:00:23.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.381+0000 7f93f54cc640 1 -- 172.21.15.60:0/1817289758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93f00736d0 msgr2=0x7f93f00b22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:23.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.381+0000 7f93f54cc640 1 --2- 172.21.15.60:0/1817289758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93f00736d0 0x7f93f00b22b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f93e4035040 tx=0x7f93e40046c0 comp rx=0 tx=0).stop 2024-05-09T05:00:23.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.381+0000 7f93f54cc640 1 -- 172.21.15.60:0/1817289758 shutdown_connections 2024-05-09T05:00:23.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.381+0000 7f93f54cc640 1 --2- 172.21.15.60:0/1817289758 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f93dc0762d0 0x7f93dc078790 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:23.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.381+0000 7f93f54cc640 1 --2- 172.21.15.60:0/1817289758 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f93f015f300 0x7f93f01613d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:23.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.381+0000 7f93f54cc640 1 --2- 172.21.15.60:0/1817289758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93f00736d0 0x7f93f00b22b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:23.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.381+0000 7f93f54cc640 1 --2- 172.21.15.60:0/1817289758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93f006fe20 0x7f93f01f9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:23.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.381+0000 7f93f54cc640 1 -- 172.21.15.60:0/1817289758 >> 172.21.15.60:0/1817289758 conn(0x7f93f00b4ef0 msgr2=0x7f93f006f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:23.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.381+0000 7f93f54cc640 1 -- 172.21.15.60:0/1817289758 shutdown_connections 2024-05-09T05:00:23.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:23.381+0000 7f93f54cc640 1 -- 172.21.15.60:0/1817289758 wait complete. 2024-05-09T05:00:23.393 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi060.front.sepia.ceph.com... 2024-05-09T05:00:23.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:23.393 DEBUG:teuthology.orchestra.run.smithi060:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-05-09T05:00:23.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi060.stderr:ceph-fuse[41956]: fuse finished with error 0 and tester_r 0 2024-05-09T05:00:23.419 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:00:29.521 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2024-05-09T05:00:29.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:29.522 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-05-09T05:00:29.554 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:00:29.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.709+0000 7f41ac075640 1 Processor -- start 2024-05-09T05:00:29.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.709+0000 7f41ac075640 1 -- start start 2024-05-09T05:00:29.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.710+0000 7f41ac075640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f41a415f300 0x7f41a41639f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:29.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.710+0000 7f41ac075640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f41a406f9f0 0x7f41a40d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:29.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.710+0000 7f41ac075640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41a40d3930 0x7f41a406ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:29.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.710+0000 7f41ac075640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f41a406ff60 con 0x7f41a415f300 2024-05-09T05:00:29.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.710+0000 7f41ac075640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f41a4070240 con 0x7f41a406f9f0 2024-05-09T05:00:29.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.710+0000 7f41ac075640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f41a4070520 con 0x7f41a40d3930 2024-05-09T05:00:29.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.710+0000 7f41a9e11640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f41a415f300 0x7f41a41639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:29.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.710+0000 7f41a9e11640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f41a415f300 0x7f41a41639f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34634/0 (socket says 172.21.15.60:34634) 2024-05-09T05:00:29.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.710+0000 7f41a9e11640 1 -- 172.21.15.60:0/2286447859 learned_addr learned my addr 172.21.15.60:0/2286447859 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:29.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.711+0000 7f41aa612640 1 --2- 172.21.15.60:0/2286447859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41a40d3930 0x7f41a406ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:29.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.711+0000 7f41a9610640 1 --2- 172.21.15.60:0/2286447859 >> v2:172.21.15.114:3300/0 conn(0x7f41a406f9f0 0x7f41a40d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:29.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.711+0000 7f41a9e11640 1 -- 172.21.15.60:0/2286447859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41a40d3930 msgr2=0x7f41a406ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:29.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.711+0000 7f41a9e11640 1 --2- 172.21.15.60:0/2286447859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41a40d3930 0x7f41a406ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:29.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.711+0000 7f41a9e11640 1 -- 172.21.15.60:0/2286447859 >> v2:172.21.15.114:3300/0 conn(0x7f41a406f9f0 msgr2=0x7f41a40d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:29.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.711+0000 7f41a9e11640 1 --2- 172.21.15.60:0/2286447859 >> v2:172.21.15.114:3300/0 conn(0x7f41a406f9f0 0x7f41a40d33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:29.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.711+0000 7f41a9e11640 1 -- 172.21.15.60:0/2286447859 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41a406e740 con 0x7f41a415f300 2024-05-09T05:00:29.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.711+0000 7f41aa612640 1 --2- 172.21.15.60:0/2286447859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41a40d3930 0x7f41a406ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:29.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.711+0000 7f41a9610640 1 --2- 172.21.15.60:0/2286447859 >> v2:172.21.15.114:3300/0 conn(0x7f41a406f9f0 0x7f41a40d33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:29.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.711+0000 7f41a9e11640 1 --2- 172.21.15.60:0/2286447859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41a415f300 0x7f41a41639f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f41a4070730 tx=0x7f41a002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3d28db13962960c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:29.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.712+0000 7f41a8e0f640 1 -- 172.21.15.60:0/2286447859 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a0037070 con 0x7f41a415f300 2024-05-09T05:00:29.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.712+0000 7f41a8e0f640 1 -- 172.21.15.60:0/2286447859 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a0032440 con 0x7f41a415f300 2024-05-09T05:00:29.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.712+0000 7f41a8e0f640 1 -- 172.21.15.60:0/2286447859 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a0032810 con 0x7f41a415f300 2024-05-09T05:00:29.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.712+0000 7f41ac075640 1 -- 172.21.15.60:0/2286447859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41a415f300 msgr2=0x7f41a41639f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:29.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.712+0000 7f41ac075640 1 --2- 172.21.15.60:0/2286447859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41a415f300 0x7f41a41639f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f41a4070730 tx=0x7f41a002a1c0 comp rx=0 tx=0).stop 2024-05-09T05:00:29.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.712+0000 7f41ac075640 1 -- 172.21.15.60:0/2286447859 shutdown_connections 2024-05-09T05:00:29.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.712+0000 7f41ac075640 1 --2- 172.21.15.60:0/2286447859 >> v2:172.21.15.114:3300/0 conn(0x7f41a406f9f0 0x7f41a40d33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:29.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.712+0000 7f41ac075640 1 --2- 172.21.15.60:0/2286447859 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41a40d3930 0x7f41a406ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:29.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.712+0000 7f41ac075640 1 --2- 172.21.15.60:0/2286447859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41a415f300 0x7f41a41639f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:29.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.712+0000 7f41ac075640 1 -- 172.21.15.60:0/2286447859 >> 172.21.15.60:0/2286447859 conn(0x7f41a409b850 msgr2=0x7f41a409bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:29.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.712+0000 7f41ac075640 1 -- 172.21.15.60:0/2286447859 shutdown_connections 2024-05-09T05:00:29.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.712+0000 7f41ac075640 1 -- 172.21.15.60:0/2286447859 wait complete. 2024-05-09T05:00:29.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.714+0000 7f41ac075640 1 Processor -- start 2024-05-09T05:00:29.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.714+0000 7f41ac075640 1 -- start start 2024-05-09T05:00:29.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.714+0000 7f41ac075640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41a406f9f0 0x7f41a4160cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:29.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.714+0000 7f41ac075640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41a40d3930 0x7f41a4161210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:29.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.714+0000 7f41ac075640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41a415f300 0x7f41a4161750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:29.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.714+0000 7f41ac075640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f41a406f480 con 0x7f41a415f300 2024-05-09T05:00:29.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.714+0000 7f41ac075640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f41a4163f30 con 0x7f41a40d3930 2024-05-09T05:00:29.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.714+0000 7f41ac075640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f41a406f600 con 0x7f41a406f9f0 2024-05-09T05:00:29.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.714+0000 7f41aa612640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41a415f300 0x7f41a4161750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:29.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.714+0000 7f41aa612640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41a415f300 0x7f41a4161750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34650/0 (socket says 172.21.15.60:34650) 2024-05-09T05:00:29.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.714+0000 7f41aa612640 1 -- 172.21.15.60:0/3968032394 learned_addr learned my addr 172.21.15.60:0/3968032394 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:29.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.715+0000 7f41a9e11640 1 --2- 172.21.15.60:0/3968032394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41a406f9f0 0x7f41a4160cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:29.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.715+0000 7f41a9610640 1 --2- 172.21.15.60:0/3968032394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41a40d3930 0x7f41a4161210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:29.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.715+0000 7f41aa612640 1 -- 172.21.15.60:0/3968032394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41a406f9f0 msgr2=0x7f41a4160cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:29.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.715+0000 7f41aa612640 1 --2- 172.21.15.60:0/3968032394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41a406f9f0 0x7f41a4160cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:29.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.715+0000 7f41aa612640 1 -- 172.21.15.60:0/3968032394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41a40d3930 msgr2=0x7f41a4161210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:29.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.715+0000 7f41aa612640 1 --2- 172.21.15.60:0/3968032394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41a40d3930 0x7f41a4161210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:29.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.715+0000 7f41aa612640 1 -- 172.21.15.60:0/3968032394 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41a40b4db0 con 0x7f41a415f300 2024-05-09T05:00:29.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.715+0000 7f41aa612640 1 --2- 172.21.15.60:0/3968032394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41a415f300 0x7f41a4161750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f419c00cef0 tx=0x7f419c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:29.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.715+0000 7f419a7fc640 1 -- 172.21.15.60:0/3968032394 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419c01a070 con 0x7f41a415f300 2024-05-09T05:00:29.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.715+0000 7f419a7fc640 1 -- 172.21.15.60:0/3968032394 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f419c00d860 con 0x7f41a415f300 2024-05-09T05:00:29.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.715+0000 7f41ac075640 1 -- 172.21.15.60:0/3968032394 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41a409ad80 con 0x7f41a415f300 2024-05-09T05:00:29.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.715+0000 7f419a7fc640 1 -- 172.21.15.60:0/3968032394 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419c00dc50 con 0x7f41a415f300 2024-05-09T05:00:29.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.715+0000 7f41ac075640 1 -- 172.21.15.60:0/3968032394 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41a4159730 con 0x7f41a415f300 2024-05-09T05:00:29.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.716+0000 7f41ac075640 1 -- 172.21.15.60:0/3968032394 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41a4051040 con 0x7f41a415f300 2024-05-09T05:00:29.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.717+0000 7f419a7fc640 1 -- 172.21.15.60:0/3968032394 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f419c01d070 con 0x7f41a415f300 2024-05-09T05:00:29.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.717+0000 7f419a7fc640 1 --2- 172.21.15.60:0/3968032394 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f41880762d0 0x7f4188078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:29.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.718+0000 7f419a7fc640 1 -- 172.21.15.60:0/3968032394 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f419c098160 con 0x7f41a415f300 2024-05-09T05:00:29.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.718+0000 7f41a9e11640 1 --2- 172.21.15.60:0/3968032394 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f41880762d0 0x7f4188078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:29.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.718+0000 7f41a9e11640 1 --2- 172.21.15.60:0/3968032394 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f41880762d0 0x7f4188078790 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f41a0034000 tx=0x7f41a002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:29.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.722+0000 7f419a7fc640 1 -- 172.21.15.60:0/3968032394 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f419c0610d0 con 0x7f41a415f300 2024-05-09T05:00:29.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.917+0000 7f41ac075640 1 -- 172.21.15.60:0/3968032394 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f41a409f8e0 con 0x7f41a415f300 2024-05-09T05:00:29.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.918+0000 7f419a7fc640 1 -- 172.21.15.60:0/3968032394 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18490 (secure 0 0 0) 0x7f419c0653d0 con 0x7f41a415f300 2024-05-09T05:00:29.919 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:29.919 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:00:17.248958+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:00:10.358133+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:00:11.157769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T05:00:14.850459+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:00:29.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.920+0000 7f41ac075640 1 -- 172.21.15.60:0/3968032394 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f41880762d0 msgr2=0x7f4188078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:29.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.920+0000 7f41ac075640 1 --2- 172.21.15.60:0/3968032394 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f41880762d0 0x7f4188078790 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f41a0034000 tx=0x7f41a002e000 comp rx=0 tx=0).stop 2024-05-09T05:00:29.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.920+0000 7f41ac075640 1 -- 172.21.15.60:0/3968032394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41a415f300 msgr2=0x7f41a4161750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:29.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.920+0000 7f41ac075640 1 --2- 172.21.15.60:0/3968032394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41a415f300 0x7f41a4161750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f419c00cef0 tx=0x7f419c00a710 comp rx=0 tx=0).stop 2024-05-09T05:00:29.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.920+0000 7f41ac075640 1 -- 172.21.15.60:0/3968032394 shutdown_connections 2024-05-09T05:00:29.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.920+0000 7f41ac075640 1 --2- 172.21.15.60:0/3968032394 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f41880762d0 0x7f4188078790 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:29.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.920+0000 7f41ac075640 1 --2- 172.21.15.60:0/3968032394 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41a415f300 0x7f41a4161750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:29.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.920+0000 7f41ac075640 1 --2- 172.21.15.60:0/3968032394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41a40d3930 0x7f41a4161210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:29.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.920+0000 7f41ac075640 1 --2- 172.21.15.60:0/3968032394 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f41a406f9f0 0x7f41a4160cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:29.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.920+0000 7f41ac075640 1 -- 172.21.15.60:0/3968032394 >> 172.21.15.60:0/3968032394 conn(0x7f41a409b850 msgr2=0x7f41a409a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:29.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.920+0000 7f41ac075640 1 -- 172.21.15.60:0/3968032394 shutdown_connections 2024-05-09T05:00:29.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:29.920+0000 7f41ac075640 1 -- 172.21.15.60:0/3968032394 wait complete. 2024-05-09T05:00:29.932 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:00:30.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.036+0000 7fea03dc1640 1 Processor -- start 2024-05-09T05:00:30.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.036+0000 7fea03dc1640 1 -- start start 2024-05-09T05:00:30.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.037+0000 7fea03dc1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe9fc070250 0x7fe9fc0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:30.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.037+0000 7fea03dc1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9fc0d2980 0x7fe9fc072850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:30.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.037+0000 7fea03dc1640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9fc06f6a0 con 0x7fe9fc15f490 2024-05-09T05:00:30.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.037+0000 7fea03dc1640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fe9fc06e8e0 con 0x7fe9fc070250 2024-05-09T05:00:30.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.037+0000 7fea03dc1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe9fc06ebc0 con 0x7fe9fc0d2980 2024-05-09T05:00:30.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.037+0000 7fea01b5d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe9fc15f490 0x7fe9fc06fb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38354/0 (socket says 172.21.15.60:38354) 2024-05-09T05:00:30.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.037+0000 7fea01b5d640 1 -- 172.21.15.60:0/3590060371 learned_addr learned my addr 172.21.15.60:0/3590060371 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:30.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.037+0000 7fea0135c640 1 --2- 172.21.15.60:0/3590060371 >> v2:172.21.15.60:3300/0 conn(0x7fe9fc070250 0x7fe9fc0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:30.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.038+0000 7fea0235e640 1 --2- 172.21.15.60:0/3590060371 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9fc0d2980 0x7fe9fc072850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:30.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.038+0000 7fea0135c640 1 -- 172.21.15.60:0/3590060371 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9fc0d2980 msgr2=0x7fe9fc072850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:30.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.038+0000 7fea0135c640 1 --2- 172.21.15.60:0/3590060371 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9fc0d2980 0x7fe9fc072850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:30.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.038+0000 7fea0135c640 1 -- 172.21.15.60:0/3590060371 >> v1:172.21.15.60:6789/0 conn(0x7fe9fc15f490 legacy=0x7fe9fc06fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:30.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.038+0000 7fea0135c640 1 -- 172.21.15.60:0/3590060371 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9fc06eea0 con 0x7fe9fc070250 2024-05-09T05:00:30.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.038+0000 7fea00b5b640 1 -- 172.21.15.60:0/3590060371 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2027409207 0 0) 0x7fe9fc06f6a0 con 0x7fe9fc15f490 2024-05-09T05:00:30.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.038+0000 7fea0135c640 1 --2- 172.21.15.60:0/3590060371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9fc070250 0x7fe9fc0d2440 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe9f00097a0 tx=0x7fe9f002a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=889a04574cc01dcf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:30.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.039+0000 7fea00b5b640 1 -- 172.21.15.60:0/3590060371 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f0037070 con 0x7fe9fc070250 2024-05-09T05:00:30.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.039+0000 7fea00b5b640 1 -- 172.21.15.60:0/3590060371 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9f002ae50 con 0x7fe9fc070250 2024-05-09T05:00:30.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.039+0000 7fea00b5b640 1 -- 172.21.15.60:0/3590060371 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f0032660 con 0x7fe9fc070250 2024-05-09T05:00:30.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.039+0000 7fea03dc1640 1 -- 172.21.15.60:0/3590060371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9fc070250 msgr2=0x7fe9fc0d2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:30.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.039+0000 7fea03dc1640 1 --2- 172.21.15.60:0/3590060371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9fc070250 0x7fe9fc0d2440 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe9f00097a0 tx=0x7fe9f002a100 comp rx=0 tx=0).stop 2024-05-09T05:00:30.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.039+0000 7fea03dc1640 1 -- 172.21.15.60:0/3590060371 shutdown_connections 2024-05-09T05:00:30.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.039+0000 7fea03dc1640 1 --2- 172.21.15.60:0/3590060371 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9fc0d2980 0x7fe9fc072850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:30.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.039+0000 7fea03dc1640 1 --2- 172.21.15.60:0/3590060371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9fc070250 0x7fe9fc0d2440 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:30.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.039+0000 7fea03dc1640 1 -- 172.21.15.60:0/3590060371 >> 172.21.15.60:0/3590060371 conn(0x7fe9fc09b8c0 msgr2=0x7fe9fc09bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:30.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.039+0000 7fea03dc1640 1 -- 172.21.15.60:0/3590060371 shutdown_connections 2024-05-09T05:00:30.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.039+0000 7fea03dc1640 1 -- 172.21.15.60:0/3590060371 wait complete. 2024-05-09T05:00:30.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.040+0000 7fea03dc1640 1 Processor -- start 2024-05-09T05:00:30.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.041+0000 7fea03dc1640 1 -- start start 2024-05-09T05:00:30.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.041+0000 7fea03dc1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9fc070250 0x7fe9fc0b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:30.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.041+0000 7fea03dc1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9fc0d2980 0x7fe9fc1e7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:30.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.041+0000 7fea03dc1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9fc15f490 0x7fe9fc0a38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:30.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.041+0000 7fea03dc1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc0d2f50 con 0x7fe9fc070250 2024-05-09T05:00:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.041+0000 7fea03dc1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc0d2dd0 con 0x7fe9fc0d2980 2024-05-09T05:00:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.041+0000 7fea03dc1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe9fc072d90 con 0x7fe9fc15f490 2024-05-09T05:00:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.041+0000 7fea01b5d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9fc070250 0x7fe9fc0b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.041+0000 7fea01b5d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9fc070250 0x7fe9fc0b4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34670/0 (socket says 172.21.15.60:34670) 2024-05-09T05:00:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.041+0000 7fea01b5d640 1 -- 172.21.15.60:0/2184886332 learned_addr learned my addr 172.21.15.60:0/2184886332 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.042+0000 7fea0235e640 1 --2- 172.21.15.60:0/2184886332 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9fc15f490 0x7fe9fc0a38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.042+0000 7fea0135c640 1 --2- 172.21.15.60:0/2184886332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9fc0d2980 0x7fe9fc1e7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.042+0000 7fea01b5d640 1 -- 172.21.15.60:0/2184886332 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9fc15f490 msgr2=0x7fe9fc0a38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.042+0000 7fea01b5d640 1 --2- 172.21.15.60:0/2184886332 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9fc15f490 0x7fe9fc0a38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.042+0000 7fea01b5d640 1 -- 172.21.15.60:0/2184886332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9fc0d2980 msgr2=0x7fe9fc1e7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.042+0000 7fea01b5d640 1 --2- 172.21.15.60:0/2184886332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9fc0d2980 0x7fe9fc1e7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.042+0000 7fea01b5d640 1 -- 172.21.15.60:0/2184886332 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9fc0b4990 con 0x7fe9fc070250 2024-05-09T05:00:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.042+0000 7fea0135c640 1 --2- 172.21.15.60:0/2184886332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9fc0d2980 0x7fe9fc1e7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:30.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.042+0000 7fea0235e640 1 --2- 172.21.15.60:0/2184886332 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9fc15f490 0x7fe9fc0a38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.042+0000 7fea01b5d640 1 --2- 172.21.15.60:0/2184886332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9fc070250 0x7fe9fc0b4450 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe9f400c410 tx=0x7fe9f400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.042+0000 7fe9fa7fc640 1 -- 172.21.15.60:0/2184886332 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f40074e0 con 0x7fe9fc070250 2024-05-09T05:00:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.042+0000 7fe9fa7fc640 1 -- 172.21.15.60:0/2184886332 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9f40047e0 con 0x7fe9fc070250 2024-05-09T05:00:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.042+0000 7fea03dc1640 1 -- 172.21.15.60:0/2184886332 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9fc1e66e0 con 0x7fe9fc070250 2024-05-09T05:00:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.042+0000 7fe9fa7fc640 1 -- 172.21.15.60:0/2184886332 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f4004bd0 con 0x7fe9fc070250 2024-05-09T05:00:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.042+0000 7fea03dc1640 1 -- 172.21.15.60:0/2184886332 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9fc1e6ca0 con 0x7fe9fc070250 2024-05-09T05:00:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.043+0000 7fea03dc1640 1 -- 172.21.15.60:0/2184886332 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9fc051040 con 0x7fe9fc070250 2024-05-09T05:00:30.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.044+0000 7fe9fa7fc640 1 -- 172.21.15.60:0/2184886332 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe9f4011070 con 0x7fe9fc070250 2024-05-09T05:00:30.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.044+0000 7fe9fa7fc640 1 --2- 172.21.15.60:0/2184886332 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe9e80762d0 0x7fe9e8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:30.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.044+0000 7fe9fa7fc640 1 -- 172.21.15.60:0/2184886332 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe9f40974b0 con 0x7fe9fc070250 2024-05-09T05:00:30.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.045+0000 7fea0135c640 1 --2- 172.21.15.60:0/2184886332 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe9e80762d0 0x7fe9e8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:30.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.045+0000 7fea0135c640 1 --2- 172.21.15.60:0/2184886332 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe9e80762d0 0x7fe9e8078790 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0034000 tx=0x7fe9f002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:30.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.049+0000 7fe9fa7fc640 1 -- 172.21.15.60:0/2184886332 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe9f4060b60 con 0x7fe9fc070250 2024-05-09T05:00:30.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.241+0000 7fea03dc1640 1 -- 172.21.15.60:0/2184886332 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe9fc099070 con 0x7fe9fc070250 2024-05-09T05:00:30.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.243+0000 7fe9fa7fc640 1 -- 172.21.15.60:0/2184886332 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18490 (secure 0 0 0) 0x7fe9f4060460 con 0x7fe9fc070250 2024-05-09T05:00:30.243 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:30.243 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:00:17.248958+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:00:10.358133+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:00:11.157769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T05:00:14.850459+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:00:30.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.245+0000 7fea03dc1640 1 -- 172.21.15.60:0/2184886332 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe9e80762d0 msgr2=0x7fe9e8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:30.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.245+0000 7fea03dc1640 1 --2- 172.21.15.60:0/2184886332 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe9e80762d0 0x7fe9e8078790 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0034000 tx=0x7fe9f002e000 comp rx=0 tx=0).stop 2024-05-09T05:00:30.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.245+0000 7fea03dc1640 1 -- 172.21.15.60:0/2184886332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9fc070250 msgr2=0x7fe9fc0b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:30.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.245+0000 7fea03dc1640 1 --2- 172.21.15.60:0/2184886332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9fc070250 0x7fe9fc0b4450 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe9f400c410 tx=0x7fe9f400c8d0 comp rx=0 tx=0).stop 2024-05-09T05:00:30.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.245+0000 7fea03dc1640 1 -- 172.21.15.60:0/2184886332 shutdown_connections 2024-05-09T05:00:30.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.245+0000 7fea03dc1640 1 --2- 172.21.15.60:0/2184886332 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe9e80762d0 0x7fe9e8078790 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:30.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.245+0000 7fea03dc1640 1 --2- 172.21.15.60:0/2184886332 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe9fc15f490 0x7fe9fc0a38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:30.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.245+0000 7fea03dc1640 1 --2- 172.21.15.60:0/2184886332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9fc0d2980 0x7fe9fc1e7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:30.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.245+0000 7fea03dc1640 1 --2- 172.21.15.60:0/2184886332 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe9fc070250 0x7fe9fc0b4450 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:30.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.245+0000 7fea03dc1640 1 -- 172.21.15.60:0/2184886332 >> 172.21.15.60:0/2184886332 conn(0x7fe9fc09b8c0 msgr2=0x7fe9fc15ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:30.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.245+0000 7fea03dc1640 1 -- 172.21.15.60:0/2184886332 shutdown_connections 2024-05-09T05:00:30.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.245+0000 7fea03dc1640 1 -- 172.21.15.60:0/2184886332 wait complete. 2024-05-09T05:00:30.257 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-09T05:00:30.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.363+0000 7fc2f83c1640 1 Processor -- start 2024-05-09T05:00:30.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.364+0000 7fc2f83c1640 1 -- start start 2024-05-09T05:00:30.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.364+0000 7fc2f83c1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2f015f3c0 0x7fc2f0162e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:30.365 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.365+0000 7fc2f83c1640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc2f00b9d00 0x7fc2f00732d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:30.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.365+0000 7fc2f83c1640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f0163740 con 0x7fc2f00b7fe0 2024-05-09T05:00:30.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.365+0000 7fc2f83c1640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fc2f0163a20 con 0x7fc2f015f3c0 2024-05-09T05:00:30.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.365+0000 7fc2f83c1640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fc2f0163d00 con 0x7fc2f00b9d00 2024-05-09T05:00:30.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.365+0000 7fc2f615d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2f015f3c0 0x7fc2f0162e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:30.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.365+0000 7fc2f615d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2f015f3c0 0x7fc2f0162e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34684/0 (socket says 172.21.15.60:34684) 2024-05-09T05:00:30.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.365+0000 7fc2f595c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc2f00b7fe0 0x7fc2f00b95f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38360/0 (socket says 172.21.15.60:38360) 2024-05-09T05:00:30.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.365+0000 7fc2f615d640 1 -- 172.21.15.60:0/1647221469 learned_addr learned my addr 172.21.15.60:0/1647221469 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:30.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.365+0000 7fc2f695e640 1 --2- 172.21.15.60:0/1647221469 >> v2:172.21.15.114:3300/0 conn(0x7fc2f00b9d00 0x7fc2f00732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:30.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.365+0000 7fc2f615d640 1 -- 172.21.15.60:0/1647221469 >> v2:172.21.15.114:3300/0 conn(0x7fc2f00b9d00 msgr2=0x7fc2f00732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:30.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.365+0000 7fc2f615d640 1 --2- 172.21.15.60:0/1647221469 >> v2:172.21.15.114:3300/0 conn(0x7fc2f00b9d00 0x7fc2f00732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:30.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.365+0000 7fc2f615d640 1 -- 172.21.15.60:0/1647221469 >> v1:172.21.15.60:6789/0 conn(0x7fc2f00b7fe0 legacy=0x7fc2f00b95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:30.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.365+0000 7fc2f615d640 1 -- 172.21.15.60:0/1647221469 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f0072ad0 con 0x7fc2f015f3c0 2024-05-09T05:00:30.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.365+0000 7fc2f695e640 1 --2- 172.21.15.60:0/1647221469 >> v2:172.21.15.114:3300/0 conn(0x7fc2f00b9d00 0x7fc2f00732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:30.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.366+0000 7fc2f615d640 1 --2- 172.21.15.60:0/1647221469 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2f015f3c0 0x7fc2f0162e40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc2f006e500 tx=0x7fc2ec02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee9066f1da6f5ecc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:30.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.366+0000 7fc2f515b640 1 -- 172.21.15.60:0/1647221469 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec037070 con 0x7fc2f015f3c0 2024-05-09T05:00:30.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.366+0000 7fc2f515b640 1 -- 172.21.15.60:0/1647221469 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2ec032440 con 0x7fc2f015f3c0 2024-05-09T05:00:30.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.366+0000 7fc2f515b640 1 -- 172.21.15.60:0/1647221469 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec032810 con 0x7fc2f015f3c0 2024-05-09T05:00:30.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.366+0000 7fc2f83c1640 1 -- 172.21.15.60:0/1647221469 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2f015f3c0 msgr2=0x7fc2f0162e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:30.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.366+0000 7fc2f83c1640 1 --2- 172.21.15.60:0/1647221469 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2f015f3c0 0x7fc2f0162e40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc2f006e500 tx=0x7fc2ec02a270 comp rx=0 tx=0).stop 2024-05-09T05:00:30.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.367+0000 7fc2f83c1640 1 -- 172.21.15.60:0/1647221469 shutdown_connections 2024-05-09T05:00:30.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.367+0000 7fc2f83c1640 1 --2- 172.21.15.60:0/1647221469 >> v2:172.21.15.114:3300/0 conn(0x7fc2f00b9d00 0x7fc2f00732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:30.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.367+0000 7fc2f83c1640 1 --2- 172.21.15.60:0/1647221469 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2f015f3c0 0x7fc2f0162e40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:30.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.367+0000 7fc2f83c1640 1 -- 172.21.15.60:0/1647221469 >> 172.21.15.60:0/1647221469 conn(0x7fc2f006cae0 msgr2=0x7fc2f00d53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:30.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.367+0000 7fc2f83c1640 1 -- 172.21.15.60:0/1647221469 shutdown_connections 2024-05-09T05:00:30.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.367+0000 7fc2f83c1640 1 -- 172.21.15.60:0/1647221469 wait complete. 2024-05-09T05:00:30.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.368+0000 7fc2f83c1640 1 Processor -- start 2024-05-09T05:00:30.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.368+0000 7fc2f83c1640 1 -- start start 2024-05-09T05:00:30.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2f83c1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2f00b7fe0 0x7fc2f0073440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:30.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2f83c1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2f00b9d00 0x7fc2f0145cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:30.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2f83c1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2f015f3c0 0x7fc2f0146200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:30.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2f83c1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f0163380 con 0x7fc2f00b9d00 2024-05-09T05:00:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2f83c1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f00d1dc0 con 0x7fc2f015f3c0 2024-05-09T05:00:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2f83c1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f0163500 con 0x7fc2f00b7fe0 2024-05-09T05:00:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2f595c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2f00b9d00 0x7fc2f0145cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2f595c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2f00b9d00 0x7fc2f0145cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34686/0 (socket says 172.21.15.60:34686) 2024-05-09T05:00:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2f595c640 1 -- 172.21.15.60:0/2693519418 learned_addr learned my addr 172.21.15.60:0/2693519418 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2f695e640 1 --2- 172.21.15.60:0/2693519418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2f015f3c0 0x7fc2f0146200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2f615d640 1 --2- 172.21.15.60:0/2693519418 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2f00b7fe0 0x7fc2f0073440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2f595c640 1 -- 172.21.15.60:0/2693519418 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2f00b7fe0 msgr2=0x7fc2f0073440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2f595c640 1 --2- 172.21.15.60:0/2693519418 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2f00b7fe0 0x7fc2f0073440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2f595c640 1 -- 172.21.15.60:0/2693519418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2f015f3c0 msgr2=0x7fc2f0146200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2f595c640 1 --2- 172.21.15.60:0/2693519418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2f015f3c0 0x7fc2f0146200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2f595c640 1 -- 172.21.15.60:0/2693519418 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f015c1d0 con 0x7fc2f00b9d00 2024-05-09T05:00:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2f595c640 1 --2- 172.21.15.60:0/2693519418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2f00b9d00 0x7fc2f0145cc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc2e000c4d0 tx=0x7fc2e000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2e6ffd640 1 -- 172.21.15.60:0/2693519418 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e0018070 con 0x7fc2f00b9d00 2024-05-09T05:00:30.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2e6ffd640 1 -- 172.21.15.60:0/2693519418 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2e0007480 con 0x7fc2f00b9d00 2024-05-09T05:00:30.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2f83c1640 1 -- 172.21.15.60:0/2693519418 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f015c510 con 0x7fc2f00b9d00 2024-05-09T05:00:30.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2e6ffd640 1 -- 172.21.15.60:0/2693519418 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e00049e0 con 0x7fc2f00b9d00 2024-05-09T05:00:30.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.369+0000 7fc2f83c1640 1 -- 172.21.15.60:0/2693519418 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f015cd50 con 0x7fc2f00b9d00 2024-05-09T05:00:30.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.370+0000 7fc2f83c1640 1 -- 172.21.15.60:0/2693519418 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f015cf50 con 0x7fc2f00b9d00 2024-05-09T05:00:30.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.371+0000 7fc2e6ffd640 1 -- 172.21.15.60:0/2693519418 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc2e001b070 con 0x7fc2f00b9d00 2024-05-09T05:00:30.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.371+0000 7fc2e6ffd640 1 --2- 172.21.15.60:0/2693519418 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc2d40762d0 0x7fc2d4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:30.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.372+0000 7fc2e6ffd640 1 -- 172.21.15.60:0/2693519418 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc2e0096c00 con 0x7fc2f00b9d00 2024-05-09T05:00:30.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.372+0000 7fc2f615d640 1 --2- 172.21.15.60:0/2693519418 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc2d40762d0 0x7fc2d4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:30.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.372+0000 7fc2f615d640 1 --2- 172.21.15.60:0/2693519418 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc2d40762d0 0x7fc2d4078790 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec034000 tx=0x7fc2ec02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:30.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.376+0000 7fc2e6ffd640 1 -- 172.21.15.60:0/2693519418 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc2f015cf50 con 0x7fc2f00b9d00 2024-05-09T05:00:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.606+0000 7fc2f83c1640 1 -- 172.21.15.60:0/2693519418 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fc2f0051040 con 0x7fc2f00b9d00 2024-05-09T05:00:30.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.978+0000 7fc2e6ffd640 1 -- 172.21.15.60:0/2693519418 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) ==== 118+0+0 (secure 0 0 0) 0x7fc2e009d080 con 0x7fc2f00b9d00 2024-05-09T05:00:30.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.980+0000 7fc2f83c1640 1 -- 172.21.15.60:0/2693519418 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc2d40762d0 msgr2=0x7fc2d4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:30.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.980+0000 7fc2f83c1640 1 --2- 172.21.15.60:0/2693519418 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc2d40762d0 0x7fc2d4078790 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec034000 tx=0x7fc2ec02e000 comp rx=0 tx=0).stop 2024-05-09T05:00:30.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.980+0000 7fc2f83c1640 1 -- 172.21.15.60:0/2693519418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2f00b9d00 msgr2=0x7fc2f0145cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:30.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.980+0000 7fc2f83c1640 1 --2- 172.21.15.60:0/2693519418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2f00b9d00 0x7fc2f0145cc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc2e000c4d0 tx=0x7fc2e000c990 comp rx=0 tx=0).stop 2024-05-09T05:00:30.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.980+0000 7fc2f83c1640 1 -- 172.21.15.60:0/2693519418 shutdown_connections 2024-05-09T05:00:30.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.980+0000 7fc2f83c1640 1 --2- 172.21.15.60:0/2693519418 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc2d40762d0 0x7fc2d4078790 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:30.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.980+0000 7fc2f83c1640 1 --2- 172.21.15.60:0/2693519418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc2f015f3c0 0x7fc2f0146200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:30.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.980+0000 7fc2f83c1640 1 --2- 172.21.15.60:0/2693519418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2f00b9d00 0x7fc2f0145cc0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:30.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.980+0000 7fc2f83c1640 1 --2- 172.21.15.60:0/2693519418 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc2f00b7fe0 0x7fc2f0073440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:30.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.980+0000 7fc2f83c1640 1 -- 172.21.15.60:0/2693519418 >> 172.21.15.60:0/2693519418 conn(0x7fc2f006cae0 msgr2=0x7fc2f015a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:30.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.980+0000 7fc2f83c1640 1 -- 172.21.15.60:0/2693519418 shutdown_connections 2024-05-09T05:00:30.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:30.980+0000 7fc2f83c1640 1 -- 172.21.15.60:0/2693519418 wait complete. 2024-05-09T05:00:30.994 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-05-09T05:00:30.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-09T05:00:31.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.102+0000 7f7092bc8640 1 Processor -- start 2024-05-09T05:00:31.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.102+0000 7f7092bc8640 1 -- start start 2024-05-09T05:00:31.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.103+0000 7f7092bc8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f708c15f330 0x7f708c0b7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:31.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.103+0000 7f7092bc8640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f708c072c00 con 0x7f708c162fa0 2024-05-09T05:00:31.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.103+0000 7f7092bc8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f708c163540 con 0x7f708c0b65d0 2024-05-09T05:00:31.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.103+0000 7f7092bc8640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f708c163820 con 0x7f708c15f330 2024-05-09T05:00:31.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.103+0000 7f7090964640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f708c15f330 0x7f708c0b7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:31.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.103+0000 7f7090964640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f708c15f330 0x7f708c0b7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34696/0 (socket says 172.21.15.60:34696) 2024-05-09T05:00:31.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.103+0000 7f7090964640 1 -- 172.21.15.60:0/1447675466 learned_addr learned my addr 172.21.15.60:0/1447675466 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:31.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.103+0000 7f7083fff640 1 --1- 172.21.15.60:0/1447675466 >> v1:172.21.15.60:6789/0 conn(0x7f708c162fa0 0x7f708c0b5ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38376/0 (socket says 172.21.15.60:38376) 2024-05-09T05:00:31.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.103+0000 7f7090964640 1 -- 172.21.15.60:0/1447675466 >> v1:172.21.15.114:6789/0 conn(0x7f708c0b65d0 legacy=0x7f708c0730e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:31.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.103+0000 7f7090964640 1 -- 172.21.15.60:0/1447675466 >> v1:172.21.15.60:6789/0 conn(0x7f708c162fa0 legacy=0x7f708c0b5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:31.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.103+0000 7f7090964640 1 -- 172.21.15.60:0/1447675466 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f708c163b00 con 0x7f708c15f330 2024-05-09T05:00:31.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.103+0000 7f70837fe640 1 -- 172.21.15.60:0/1447675466 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2062347900 0 0) 0x7f708c072c00 con 0x7f708c162fa0 2024-05-09T05:00:31.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.104+0000 7f7090964640 1 --2- 172.21.15.60:0/1447675466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f708c15f330 0x7f708c0b7e20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7088009870 tx=0x7f708802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=650a9ea1243fed6d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:31.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.104+0000 7f70837fe640 1 -- 172.21.15.60:0/1447675466 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7088037070 con 0x7f708c15f330 2024-05-09T05:00:31.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.104+0000 7f70837fe640 1 -- 172.21.15.60:0/1447675466 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7088032440 con 0x7f708c15f330 2024-05-09T05:00:31.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.104+0000 7f70837fe640 1 -- 172.21.15.60:0/1447675466 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7088032810 con 0x7f708c15f330 2024-05-09T05:00:31.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.104+0000 7f7092bc8640 1 -- 172.21.15.60:0/1447675466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f708c15f330 msgr2=0x7f708c0b7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:31.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.104+0000 7f7092bc8640 1 --2- 172.21.15.60:0/1447675466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f708c15f330 0x7f708c0b7e20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7088009870 tx=0x7f708802a270 comp rx=0 tx=0).stop 2024-05-09T05:00:31.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.105+0000 7f7092bc8640 1 -- 172.21.15.60:0/1447675466 shutdown_connections 2024-05-09T05:00:31.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.105+0000 7f7092bc8640 1 --2- 172.21.15.60:0/1447675466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f708c15f330 0x7f708c0b7e20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:31.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.105+0000 7f7092bc8640 1 -- 172.21.15.60:0/1447675466 >> 172.21.15.60:0/1447675466 conn(0x7f708c06ca80 msgr2=0x7f708c0d53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:31.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.105+0000 7f7092bc8640 1 -- 172.21.15.60:0/1447675466 shutdown_connections 2024-05-09T05:00:31.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.105+0000 7f7092bc8640 1 -- 172.21.15.60:0/1447675466 wait complete. 2024-05-09T05:00:31.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.106+0000 7f7092bc8640 1 Processor -- start 2024-05-09T05:00:31.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.106+0000 7f7092bc8640 1 -- start start 2024-05-09T05:00:31.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f7092bc8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f708c0b65d0 0x7f708c0d2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:31.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f7092bc8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f708c15f330 0x7f708c14efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:31.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f7092bc8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f708c162fa0 0x7f708c151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:31.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f7092bc8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f708c15f710 con 0x7f708c0b65d0 2024-05-09T05:00:31.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f7092bc8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f708c163de0 con 0x7f708c162fa0 2024-05-09T05:00:31.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f7092bc8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f708c15b7f0 con 0x7f708c15f330 2024-05-09T05:00:31.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f7090964640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f708c0b65d0 0x7f708c0d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:31.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f7090964640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f708c0b65d0 0x7f708c0d2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34698/0 (socket says 172.21.15.60:34698) 2024-05-09T05:00:31.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f7090964640 1 -- 172.21.15.60:0/1259918004 learned_addr learned my addr 172.21.15.60:0/1259918004 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:31.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f7091165640 1 --2- 172.21.15.60:0/1259918004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f708c162fa0 0x7f708c151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:31.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f7083fff640 1 --2- 172.21.15.60:0/1259918004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f708c15f330 0x7f708c14efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:31.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f7090964640 1 -- 172.21.15.60:0/1259918004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f708c15f330 msgr2=0x7f708c14efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:31.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f7090964640 1 --2- 172.21.15.60:0/1259918004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f708c15f330 0x7f708c14efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:31.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f7090964640 1 -- 172.21.15.60:0/1259918004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f708c162fa0 msgr2=0x7f708c151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:31.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f7090964640 1 --2- 172.21.15.60:0/1259918004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f708c162fa0 0x7f708c151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:31.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f7090964640 1 -- 172.21.15.60:0/1259918004 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f708c14f4e0 con 0x7f708c0b65d0 2024-05-09T05:00:31.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f7083fff640 1 --2- 172.21.15.60:0/1259918004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f708c15f330 0x7f708c14efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:31.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f7090964640 1 --2- 172.21.15.60:0/1259918004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f708c0b65d0 0x7f708c0d2890 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7088009870 tx=0x7f70880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:31.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f70817fa640 1 -- 172.21.15.60:0/1259918004 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7088041070 con 0x7f708c0b65d0 2024-05-09T05:00:31.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.107+0000 7f70817fa640 1 -- 172.21.15.60:0/1259918004 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708803c870 con 0x7f708c0b65d0 2024-05-09T05:00:31.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.108+0000 7f70817fa640 1 -- 172.21.15.60:0/1259918004 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7088037040 con 0x7f708c0b65d0 2024-05-09T05:00:31.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.108+0000 7f7092bc8640 1 -- 172.21.15.60:0/1259918004 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f708c0d2f50 con 0x7f708c0b65d0 2024-05-09T05:00:31.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.108+0000 7f7092bc8640 1 -- 172.21.15.60:0/1259918004 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f708c0d3510 con 0x7f708c0b65d0 2024-05-09T05:00:31.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.109+0000 7f7092bc8640 1 -- 172.21.15.60:0/1259918004 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f708c051040 con 0x7f708c0b65d0 2024-05-09T05:00:31.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.109+0000 7f70817fa640 1 -- 172.21.15.60:0/1259918004 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7088044070 con 0x7f708c0b65d0 2024-05-09T05:00:31.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.110+0000 7f70817fa640 1 --2- 172.21.15.60:0/1259918004 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f70780762d0 0x7f7078078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:31.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.110+0000 7f70817fa640 1 -- 172.21.15.60:0/1259918004 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f70880b8370 con 0x7f708c0b65d0 2024-05-09T05:00:31.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.110+0000 7f7083fff640 1 --2- 172.21.15.60:0/1259918004 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f70780762d0 0x7f7078078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:31.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.111+0000 7f7083fff640 1 --2- 172.21.15.60:0/1259918004 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f70780762d0 0x7f7078078790 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f708c155c10 tx=0x7f707c007bf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:31.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.114+0000 7f70817fa640 1 -- 172.21.15.60:0/1259918004 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7088081340 con 0x7f708c0b65d0 2024-05-09T05:00:31.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.345+0000 7f7092bc8640 1 -- 172.21.15.60:0/1259918004 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f708c0ba950 con 0x7f708c0b65d0 2024-05-09T05:00:31.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.979+0000 7f70817fa640 1 -- 172.21.15.60:0/1259918004 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) ==== 122+0+0 (secure 0 0 0) 0x7f7088085640 con 0x7f708c0b65d0 2024-05-09T05:00:31.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.982+0000 7f7092bc8640 1 -- 172.21.15.60:0/1259918004 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f70780762d0 msgr2=0x7f7078078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:31.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.982+0000 7f7092bc8640 1 --2- 172.21.15.60:0/1259918004 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f70780762d0 0x7f7078078790 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f708c155c10 tx=0x7f707c007bf0 comp rx=0 tx=0).stop 2024-05-09T05:00:31.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.982+0000 7f7092bc8640 1 -- 172.21.15.60:0/1259918004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f708c0b65d0 msgr2=0x7f708c0d2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:31.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.982+0000 7f7092bc8640 1 --2- 172.21.15.60:0/1259918004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f708c0b65d0 0x7f708c0d2890 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7088009870 tx=0x7f70880046c0 comp rx=0 tx=0).stop 2024-05-09T05:00:31.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.982+0000 7f7092bc8640 1 -- 172.21.15.60:0/1259918004 shutdown_connections 2024-05-09T05:00:31.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.982+0000 7f7092bc8640 1 --2- 172.21.15.60:0/1259918004 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f70780762d0 0x7f7078078790 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:31.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.982+0000 7f7092bc8640 1 --2- 172.21.15.60:0/1259918004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f708c162fa0 0x7f708c151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:31.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.982+0000 7f7092bc8640 1 --2- 172.21.15.60:0/1259918004 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f708c15f330 0x7f708c14efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:31.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.982+0000 7f7092bc8640 1 --2- 172.21.15.60:0/1259918004 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f708c0b65d0 0x7f708c0d2890 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:31.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.982+0000 7f7092bc8640 1 -- 172.21.15.60:0/1259918004 >> 172.21.15.60:0/1259918004 conn(0x7f708c06ca80 msgr2=0x7f708c0b56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:31.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.982+0000 7f7092bc8640 1 -- 172.21.15.60:0/1259918004 shutdown_connections 2024-05-09T05:00:31.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:31.982+0000 7f7092bc8640 1 -- 172.21.15.60:0/1259918004 wait complete. 2024-05-09T05:00:31.997 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-09T05:00:31.997 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:00:32.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.106+0000 7f30681da640 1 Processor -- start 2024-05-09T05:00:32.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.106+0000 7f30681da640 1 -- start start 2024-05-09T05:00:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.107+0000 7f30681da640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f306015f300 0x7f30601639f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.107+0000 7f30681da640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f306006f9f0 0x7f30600d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.107+0000 7f30681da640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30600d3930 0x7f306006ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.107+0000 7f30681da640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f306006ff60 con 0x7f306006f9f0 2024-05-09T05:00:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.107+0000 7f30681da640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f3060070240 con 0x7f306015f300 2024-05-09T05:00:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.107+0000 7f30681da640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f3060070520 con 0x7f30600d3930 2024-05-09T05:00:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.107+0000 7f3065775640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f306006f9f0 0x7f30600d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.107+0000 7f3065775640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f306006f9f0 0x7f30600d33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34706/0 (socket says 172.21.15.60:34706) 2024-05-09T05:00:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.107+0000 7f3065775640 1 -- 172.21.15.60:0/3343240678 learned_addr learned my addr 172.21.15.60:0/3343240678 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:32.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.107+0000 7f3065f76640 1 --2- 172.21.15.60:0/3343240678 >> v2:172.21.15.114:3300/0 conn(0x7f306015f300 0x7f30601639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:32.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.107+0000 7f3065775640 1 -- 172.21.15.60:0/3343240678 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30600d3930 msgr2=0x7f306006ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:32.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.107+0000 7f3065775640 1 --2- 172.21.15.60:0/3343240678 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30600d3930 0x7f306006ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.107+0000 7f3065775640 1 -- 172.21.15.60:0/3343240678 >> v2:172.21.15.114:3300/0 conn(0x7f306015f300 msgr2=0x7f30601639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:32.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.107+0000 7f3065775640 1 --2- 172.21.15.60:0/3343240678 >> v2:172.21.15.114:3300/0 conn(0x7f306015f300 0x7f30601639f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.107+0000 7f3065775640 1 -- 172.21.15.60:0/3343240678 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f306006e740 con 0x7f306006f9f0 2024-05-09T05:00:32.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.108+0000 7f3065775640 1 --2- 172.21.15.60:0/3343240678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f306006f9f0 0x7f30600d33f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3060070730 tx=0x7f305002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=559a857f50fc33c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:32.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.108+0000 7f3064f74640 1 -- 172.21.15.60:0/3343240678 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3050037070 con 0x7f306006f9f0 2024-05-09T05:00:32.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.108+0000 7f3064f74640 1 -- 172.21.15.60:0/3343240678 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3050032440 con 0x7f306006f9f0 2024-05-09T05:00:32.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.108+0000 7f3064f74640 1 -- 172.21.15.60:0/3343240678 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3050032810 con 0x7f306006f9f0 2024-05-09T05:00:32.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.109+0000 7f30681da640 1 -- 172.21.15.60:0/3343240678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f306006f9f0 msgr2=0x7f30600d33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:32.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.109+0000 7f30681da640 1 --2- 172.21.15.60:0/3343240678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f306006f9f0 0x7f30600d33f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3060070730 tx=0x7f305002a1c0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.109+0000 7f30681da640 1 -- 172.21.15.60:0/3343240678 shutdown_connections 2024-05-09T05:00:32.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.109+0000 7f30681da640 1 --2- 172.21.15.60:0/3343240678 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30600d3930 0x7f306006ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.109+0000 7f30681da640 1 --2- 172.21.15.60:0/3343240678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f306006f9f0 0x7f30600d33f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.109+0000 7f30681da640 1 --2- 172.21.15.60:0/3343240678 >> v2:172.21.15.114:3300/0 conn(0x7f306015f300 0x7f30601639f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.109+0000 7f30681da640 1 -- 172.21.15.60:0/3343240678 >> 172.21.15.60:0/3343240678 conn(0x7f306009b850 msgr2=0x7f306009bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:32.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.109+0000 7f30681da640 1 -- 172.21.15.60:0/3343240678 shutdown_connections 2024-05-09T05:00:32.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.109+0000 7f30681da640 1 -- 172.21.15.60:0/3343240678 wait complete. 2024-05-09T05:00:32.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.110+0000 7f30681da640 1 Processor -- start 2024-05-09T05:00:32.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.110+0000 7f30681da640 1 -- start start 2024-05-09T05:00:32.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.111+0000 7f30681da640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30600d3930 0x7f30601580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.111+0000 7f30681da640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f306015f300 0x7f30600ae390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.111+0000 7f30681da640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30600ae8d0 0x7f30601e97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.111+0000 7f30681da640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f306006f480 con 0x7f306015f300 2024-05-09T05:00:32.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.111+0000 7f30681da640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3060163f30 con 0x7f30600ae8d0 2024-05-09T05:00:32.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.111+0000 7f30681da640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f306006f600 con 0x7f30600d3930 2024-05-09T05:00:32.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.111+0000 7f3065775640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f306015f300 0x7f30600ae390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:32.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.111+0000 7f3065775640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f306015f300 0x7f30600ae390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34722/0 (socket says 172.21.15.60:34722) 2024-05-09T05:00:32.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.111+0000 7f3065775640 1 -- 172.21.15.60:0/3454157596 learned_addr learned my addr 172.21.15.60:0/3454157596 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:32.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.111+0000 7f3066777640 1 --2- 172.21.15.60:0/3454157596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30600ae8d0 0x7f30601e97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:32.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.111+0000 7f3065f76640 1 --2- 172.21.15.60:0/3454157596 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30600d3930 0x7f30601580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:32.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.111+0000 7f3065775640 1 -- 172.21.15.60:0/3454157596 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30600d3930 msgr2=0x7f30601580b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:32.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.111+0000 7f3065775640 1 --2- 172.21.15.60:0/3454157596 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30600d3930 0x7f30601580b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.111+0000 7f3065775640 1 -- 172.21.15.60:0/3454157596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30600ae8d0 msgr2=0x7f30601e97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:32.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.111+0000 7f3065775640 1 --2- 172.21.15.60:0/3454157596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30600ae8d0 0x7f30601e97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.111+0000 7f3065775640 1 -- 172.21.15.60:0/3454157596 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3060160de0 con 0x7f306015f300 2024-05-09T05:00:32.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.112+0000 7f3065775640 1 --2- 172.21.15.60:0/3454157596 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f306015f300 0x7f30600ae390 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3050035040 tx=0x7f30500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:32.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.112+0000 7f30567fc640 1 -- 172.21.15.60:0/3454157596 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3050041070 con 0x7f306015f300 2024-05-09T05:00:32.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.112+0000 7f30567fc640 1 -- 172.21.15.60:0/3454157596 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3050004220 con 0x7f306015f300 2024-05-09T05:00:32.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.112+0000 7f30567fc640 1 -- 172.21.15.60:0/3454157596 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3050037040 con 0x7f306015f300 2024-05-09T05:00:32.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.112+0000 7f30681da640 1 -- 172.21.15.60:0/3454157596 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30601610c0 con 0x7f306015f300 2024-05-09T05:00:32.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.112+0000 7f30681da640 1 -- 172.21.15.60:0/3454157596 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30601585f0 con 0x7f306015f300 2024-05-09T05:00:32.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.113+0000 7f30681da640 1 -- 172.21.15.60:0/3454157596 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3060163c90 con 0x7f306015f300 2024-05-09T05:00:32.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.113+0000 7f30567fc640 1 -- 172.21.15.60:0/3454157596 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f305002ad10 con 0x7f306015f300 2024-05-09T05:00:32.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.114+0000 7f30567fc640 1 --2- 172.21.15.60:0/3454157596 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f30440762d0 0x7f3044078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.114+0000 7f30567fc640 1 -- 172.21.15.60:0/3454157596 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f30500b7dc0 con 0x7f306015f300 2024-05-09T05:00:32.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.114+0000 7f3065f76640 1 --2- 172.21.15.60:0/3454157596 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f30440762d0 0x7f3044078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.114+0000 7f3065f76640 1 --2- 172.21.15.60:0/3454157596 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f30440762d0 0x7f3044078790 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f30601ea2b0 tx=0x7f305c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.118+0000 7f30567fc640 1 -- 172.21.15.60:0/3454157596 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3060163c90 con 0x7f306015f300 2024-05-09T05:00:32.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.346+0000 7f30681da640 1 -- 172.21.15.60:0/3454157596 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3060051040 con 0x7f306015f300 2024-05-09T05:00:32.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.347+0000 7f30567fc640 1 -- 172.21.15.60:0/3454157596 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7f3050080d90 con 0x7f306015f300 2024-05-09T05:00:32.347 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:32.348 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1285340937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1285340937},{"type":"v1","addr":"172.21.15.60:6833","nonce":1285340937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2599193390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2599193390},{"type":"v1","addr":"172.21.15.114:6835","nonce":2599193390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3432609307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3432609307},{"type":"v1","addr":"172.21.15.114:6837","nonce":3432609307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:00:12.208556+0000","modified":"2024-05-09T05:00:31.347095+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2135423314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2135423314},{"type":"v1","addr":"172.21.15.60:6835","nonce":2135423314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-05-09T05:00:32.348 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2024-05-09T05:00:32.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.349+0000 7f30681da640 1 -- 172.21.15.60:0/3454157596 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f30440762d0 msgr2=0x7f3044078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:32.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.349+0000 7f30681da640 1 --2- 172.21.15.60:0/3454157596 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f30440762d0 0x7f3044078790 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f30601ea2b0 tx=0x7f305c008040 comp rx=0 tx=0).stop 2024-05-09T05:00:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.349+0000 7f30681da640 1 -- 172.21.15.60:0/3454157596 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f306015f300 msgr2=0x7f30600ae390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.349+0000 7f30681da640 1 --2- 172.21.15.60:0/3454157596 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f306015f300 0x7f30600ae390 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3050035040 tx=0x7f30500046c0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.349+0000 7f30681da640 1 -- 172.21.15.60:0/3454157596 shutdown_connections 2024-05-09T05:00:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.349+0000 7f30681da640 1 --2- 172.21.15.60:0/3454157596 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f30440762d0 0x7f3044078790 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.349+0000 7f30681da640 1 --2- 172.21.15.60:0/3454157596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30600ae8d0 0x7f30601e97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.349+0000 7f30681da640 1 --2- 172.21.15.60:0/3454157596 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f306015f300 0x7f30600ae390 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.349+0000 7f30681da640 1 --2- 172.21.15.60:0/3454157596 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f30600d3930 0x7f30601580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.349+0000 7f30681da640 1 -- 172.21.15.60:0/3454157596 >> 172.21.15.60:0/3454157596 conn(0x7f306009b850 msgr2=0x7f3060162b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.349+0000 7f30681da640 1 -- 172.21.15.60:0/3454157596 shutdown_connections 2024-05-09T05:00:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.349+0000 7f30681da640 1 -- 172.21.15.60:0/3454157596 wait complete. 2024-05-09T05:00:32.360 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:00:32.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.463+0000 7f0bd8baf640 1 Processor -- start 2024-05-09T05:00:32.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.463+0000 7f0bd8baf640 1 -- start start 2024-05-09T05:00:32.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.464+0000 7f0bd8baf640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0bd415f300 0x7f0bd4072220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.464+0000 7f0bd8baf640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bd4072760 con 0x7f0bd4073de0 2024-05-09T05:00:32.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.464+0000 7f0bd8baf640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bd40d08e0 con 0x7f0bd406fe20 2024-05-09T05:00:32.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.464+0000 7f0bd8baf640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f0bd4073240 con 0x7f0bd415f300 2024-05-09T05:00:32.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.464+0000 7f0bd2d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0bd4073de0 0x7f0bd415f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38382/0 (socket says 172.21.15.60:38382) 2024-05-09T05:00:32.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.464+0000 7f0bd2d9d640 1 -- 172.21.15.60:0/4118840368 learned_addr learned my addr 172.21.15.60:0/4118840368 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:32.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd259c640 1 --2- 172.21.15.60:0/4118840368 >> v2:172.21.15.114:3300/0 conn(0x7f0bd415f300 0x7f0bd4072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:32.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd159a640 1 -- 172.21.15.60:0/4118840368 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2173458938 0 0) 0x7f0bc4001490 con 0x7f0bd4073de0 2024-05-09T05:00:32.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd159a640 1 -- 172.21.15.60:0/4118840368 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bc0003520 con 0x7f0bd4073de0 2024-05-09T05:00:32.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd159a640 1 -- 172.21.15.60:0/4118840368 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 864026340 0 0) 0x7f0bc0003520 con 0x7f0bd4073de0 2024-05-09T05:00:32.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd159a640 1 -- 172.21.15.60:0/4118840368 >> v2:172.21.15.114:3300/0 conn(0x7f0bd415f300 msgr2=0x7f0bd4072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:32.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd159a640 1 --2- 172.21.15.60:0/4118840368 >> v2:172.21.15.114:3300/0 conn(0x7f0bd415f300 0x7f0bd4072220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd159a640 1 -- 172.21.15.60:0/4118840368 >> v1:172.21.15.114:6789/0 conn(0x7f0bd406fe20 legacy=0x7f0bd40736d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:32.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd159a640 1 -- 172.21.15.60:0/4118840368 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bd40d0bc0 con 0x7f0bd4073de0 2024-05-09T05:00:32.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd159a640 1 -- 172.21.15.60:0/4118840368 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f0bc4001c40 con 0x7f0bd4073de0 2024-05-09T05:00:32.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd159a640 1 -- 172.21.15.60:0/4118840368 >> v1:172.21.15.60:6789/0 conn(0x7f0bd4073de0 legacy=0x7f0bd415f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:32.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd159a640 1 --2- 172.21.15.60:0/4118840368 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bc0003a50 0x7f0bc0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd159a640 1 --2- 172.21.15.60:0/4118840368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc0004800 0x7f0bc0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd159a640 1 --2- 172.21.15.60:0/4118840368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc00051c0 0x7f0bc000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd159a640 1 -- 172.21.15.60:0/4118840368 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd4073240 con 0x7f0bc0004800 2024-05-09T05:00:32.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd159a640 1 -- 172.21.15.60:0/4118840368 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc000c160 con 0x7f0bc00051c0 2024-05-09T05:00:32.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd159a640 1 -- 172.21.15.60:0/4118840368 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f0bc000c360 con 0x7f0bc0003a50 2024-05-09T05:00:32.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd1d9b640 1 --2- 172.21.15.60:0/4118840368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc0004800 0x7f0bc0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:32.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd259c640 1 --2- 172.21.15.60:0/4118840368 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bc0003a50 0x7f0bc0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:32.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd2d9d640 1 --2- 172.21.15.60:0/4118840368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc00051c0 0x7f0bc000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:32.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd1d9b640 1 -- 172.21.15.60:0/4118840368 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bc0003a50 msgr2=0x7f0bc0003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:32.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.465+0000 7f0bd1d9b640 1 --2- 172.21.15.60:0/4118840368 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bc0003a50 0x7f0bc0003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.466+0000 7f0bd1d9b640 1 -- 172.21.15.60:0/4118840368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc00051c0 msgr2=0x7f0bc000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:32.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.466+0000 7f0bd1d9b640 1 --2- 172.21.15.60:0/4118840368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc00051c0 0x7f0bc000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.466+0000 7f0bd1d9b640 1 -- 172.21.15.60:0/4118840368 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bc0003520 con 0x7f0bc0004800 2024-05-09T05:00:32.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.466+0000 7f0bd2d9d640 1 -- 172.21.15.60:0/4118840368 reap_dead start 2024-05-09T05:00:32.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.466+0000 7f0bd1d9b640 1 --2- 172.21.15.60:0/4118840368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc0004800 0x7f0bc0004c80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc001790 tx=0x7f0bc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=510d49b93a925436 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:32.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.466+0000 7f0bd159a640 1 -- 172.21.15.60:0/4118840368 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc0013b0 con 0x7f0bc0004800 2024-05-09T05:00:32.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.466+0000 7f0bd159a640 1 -- 172.21.15.60:0/4118840368 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bbc0052d0 con 0x7f0bc0004800 2024-05-09T05:00:32.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.466+0000 7f0bd159a640 1 -- 172.21.15.60:0/4118840368 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc02aac0 con 0x7f0bc0004800 2024-05-09T05:00:32.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.467+0000 7f0bd8baf640 1 -- 172.21.15.60:0/4118840368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc0004800 msgr2=0x7f0bc0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:32.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.467+0000 7f0bd8baf640 1 --2- 172.21.15.60:0/4118840368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc0004800 0x7f0bc0004c80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc001790 tx=0x7f0bc0005830 comp rx=0 tx=0).stop 2024-05-09T05:00:32.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.467+0000 7f0bd8baf640 1 -- 172.21.15.60:0/4118840368 shutdown_connections 2024-05-09T05:00:32.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.467+0000 7f0bd8baf640 1 --2- 172.21.15.60:0/4118840368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc0004800 0x7f0bc0004c80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.467+0000 7f0bd8baf640 1 -- 172.21.15.60:0/4118840368 >> 172.21.15.60:0/4118840368 conn(0x7f0bd40b4ef0 msgr2=0x7f0bd406e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:32.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.467+0000 7f0bd8baf640 1 -- 172.21.15.60:0/4118840368 shutdown_connections 2024-05-09T05:00:32.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.467+0000 7f0bd8baf640 1 -- 172.21.15.60:0/4118840368 wait complete. 2024-05-09T05:00:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.468+0000 7f0bd8baf640 1 Processor -- start 2024-05-09T05:00:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.468+0000 7f0bd8baf640 1 -- start start 2024-05-09T05:00:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.469+0000 7f0bd8baf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc0004800 0x7f0bd415c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.469+0000 7f0bd8baf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bd40d3e60 0x7f0bd40d42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.469+0000 7f0bd8baf640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bd40aee30 0x7f0bd40af2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.469+0000 7f0bd8baf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd40d04f0 con 0x7f0bd40d3e60 2024-05-09T05:00:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.469+0000 7f0bd8baf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd409c1e0 con 0x7f0bc0004800 2024-05-09T05:00:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.469+0000 7f0bd8baf640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f0bd40d0670 con 0x7f0bd40aee30 2024-05-09T05:00:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.469+0000 7f0bd1d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bd40d3e60 0x7f0bd40d42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.469+0000 7f0bd1d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bd40d3e60 0x7f0bd40d42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34748/0 (socket says 172.21.15.60:34748) 2024-05-09T05:00:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.469+0000 7f0bd1d9b640 1 -- 172.21.15.60:0/617845336 learned_addr learned my addr 172.21.15.60:0/617845336 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.469+0000 7f0bd259c640 1 --2- 172.21.15.60:0/617845336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc0004800 0x7f0bd415c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.469+0000 7f0bd2d9d640 1 --2- 172.21.15.60:0/617845336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bd40aee30 0x7f0bd40af2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.470+0000 7f0bd1d9b640 1 -- 172.21.15.60:0/617845336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bd40aee30 msgr2=0x7f0bd40af2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.470+0000 7f0bd1d9b640 1 --2- 172.21.15.60:0/617845336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bd40aee30 0x7f0bd40af2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.470+0000 7f0bd1d9b640 1 -- 172.21.15.60:0/617845336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc0004800 msgr2=0x7f0bd415c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.470+0000 7f0bd1d9b640 1 --2- 172.21.15.60:0/617845336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc0004800 0x7f0bd415c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.470+0000 7f0bd1d9b640 1 -- 172.21.15.60:0/617845336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bd40add90 con 0x7f0bd40d3e60 2024-05-09T05:00:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.470+0000 7f0bd2d9d640 1 --2- 172.21.15.60:0/617845336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bd40aee30 0x7f0bd40af2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.470+0000 7f0bd1d9b640 1 --2- 172.21.15.60:0/617845336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bd40d3e60 0x7f0bd40d42e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc001790 tx=0x7f0bbc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.470+0000 7f0bcaffd640 1 -- 172.21.15.60:0/617845336 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc03f020 con 0x7f0bd40d3e60 2024-05-09T05:00:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.470+0000 7f0bcaffd640 1 -- 172.21.15.60:0/617845336 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bbc005480 con 0x7f0bd40d3e60 2024-05-09T05:00:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.470+0000 7f0bd8baf640 1 -- 172.21.15.60:0/617845336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bd40ae070 con 0x7f0bd40d3e60 2024-05-09T05:00:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.470+0000 7f0bcaffd640 1 -- 172.21.15.60:0/617845336 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc037040 con 0x7f0bd40d3e60 2024-05-09T05:00:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.470+0000 7f0bd8baf640 1 -- 172.21.15.60:0/617845336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bd40ae630 con 0x7f0bd40d3e60 2024-05-09T05:00:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.471+0000 7f0bd8baf640 1 -- 172.21.15.60:0/617845336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bd4051040 con 0x7f0bd40d3e60 2024-05-09T05:00:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.472+0000 7f0bcaffd640 1 -- 172.21.15.60:0/617845336 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0bbc042020 con 0x7f0bd40d3e60 2024-05-09T05:00:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.472+0000 7f0bcaffd640 1 --2- 172.21.15.60:0/617845336 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0bb4076300 0x7f0bb40787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.472+0000 7f0bcaffd640 1 -- 172.21.15.60:0/617845336 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0bbc0b72f0 con 0x7f0bd40d3e60 2024-05-09T05:00:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.472+0000 7f0bd259c640 1 --2- 172.21.15.60:0/617845336 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0bb4076300 0x7f0bb40787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.473+0000 7f0bd259c640 1 --2- 172.21.15.60:0/617845336 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0bb4076300 0x7f0bb40787c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc009c40 tx=0x7f0bcc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.477+0000 7f0bcaffd640 1 -- 172.21.15.60:0/617845336 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0bbc049030 con 0x7f0bd40d3e60 2024-05-09T05:00:32.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.668+0000 7f0bd8baf640 1 -- 172.21.15.60:0/617845336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0bd40694a0 con 0x7f0bd40d3e60 2024-05-09T05:00:32.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.669+0000 7f0bcaffd640 1 -- 172.21.15.60:0/617845336 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18490 (secure 0 0 0) 0x7f0bbc0809a0 con 0x7f0bd40d3e60 2024-05-09T05:00:32.670 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:32.670 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:00:17.248958+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:00:10.358133+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:00:11.157769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T05:00:14.850459+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:00:32.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.671+0000 7f0bd8baf640 1 -- 172.21.15.60:0/617845336 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0bb4076300 msgr2=0x7f0bb40787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:32.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.671+0000 7f0bd8baf640 1 --2- 172.21.15.60:0/617845336 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0bb4076300 0x7f0bb40787c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc009c40 tx=0x7f0bcc009190 comp rx=0 tx=0).stop 2024-05-09T05:00:32.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.671+0000 7f0bd8baf640 1 -- 172.21.15.60:0/617845336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bd40d3e60 msgr2=0x7f0bd40d42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:32.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.671+0000 7f0bd8baf640 1 --2- 172.21.15.60:0/617845336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bd40d3e60 0x7f0bd40d42e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc001790 tx=0x7f0bbc001120 comp rx=0 tx=0).stop 2024-05-09T05:00:32.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.671+0000 7f0bd8baf640 1 -- 172.21.15.60:0/617845336 shutdown_connections 2024-05-09T05:00:32.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.671+0000 7f0bd8baf640 1 --2- 172.21.15.60:0/617845336 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0bb4076300 0x7f0bb40787c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.671+0000 7f0bd8baf640 1 --2- 172.21.15.60:0/617845336 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0bd40aee30 0x7f0bd40af2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.671+0000 7f0bd8baf640 1 --2- 172.21.15.60:0/617845336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bd40d3e60 0x7f0bd40d42e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.671+0000 7f0bd8baf640 1 --2- 172.21.15.60:0/617845336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bc0004800 0x7f0bd415c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.671+0000 7f0bd8baf640 1 -- 172.21.15.60:0/617845336 >> 172.21.15.60:0/617845336 conn(0x7f0bd40b4ef0 msgr2=0x7f0bd415f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:32.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.671+0000 7f0bd8baf640 1 -- 172.21.15.60:0/617845336 shutdown_connections 2024-05-09T05:00:32.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.671+0000 7f0bd8baf640 1 -- 172.21.15.60:0/617845336 wait complete. 2024-05-09T05:00:32.683 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-09T05:00:32.684 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:00:32.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.786+0000 7f62f2c99640 1 Processor -- start 2024-05-09T05:00:32.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.786+0000 7f62f2c99640 1 -- start start 2024-05-09T05:00:32.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.787+0000 7f62f2c99640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f62ec15f300 0x7f62ec1639f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.787+0000 7f62f2c99640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62ec06ff60 con 0x7f62ec06f9f0 2024-05-09T05:00:32.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.787+0000 7f62f2c99640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62ec070240 con 0x7f62ec1606f0 2024-05-09T05:00:32.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.787+0000 7f62f2c99640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f62ec070520 con 0x7f62ec15f300 2024-05-09T05:00:32.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.787+0000 7f62ebfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f62ec06f9f0 0x7f62ec0d33f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38392/0 (socket says 172.21.15.60:38392) 2024-05-09T05:00:32.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.787+0000 7f62ebfff640 1 -- 172.21.15.60:0/3266478835 learned_addr learned my addr 172.21.15.60:0/3266478835 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:32.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.788+0000 7f62f0a35640 1 --2- 172.21.15.60:0/3266478835 >> v2:172.21.15.114:3300/0 conn(0x7f62ec15f300 0x7f62ec1639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:32.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.788+0000 7f62eb7fe640 1 -- 172.21.15.60:0/3266478835 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3893661009 0 0) 0x7f62ec06ff60 con 0x7f62ec06f9f0 2024-05-09T05:00:32.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.788+0000 7f62eb7fe640 1 -- 172.21.15.60:0/3266478835 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62d8003520 con 0x7f62ec06f9f0 2024-05-09T05:00:32.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.788+0000 7f62f0a35640 1 -- 172.21.15.60:0/3266478835 >> v1:172.21.15.114:6789/0 conn(0x7f62ec1606f0 legacy=0x7f62ec06ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:32.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.788+0000 7f62f0a35640 1 -- 172.21.15.60:0/3266478835 >> v1:172.21.15.60:6789/0 conn(0x7f62ec06f9f0 legacy=0x7f62ec0d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:32.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.788+0000 7f62f0a35640 1 -- 172.21.15.60:0/3266478835 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec06e740 con 0x7f62ec15f300 2024-05-09T05:00:32.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.789+0000 7f62f0a35640 1 --2- 172.21.15.60:0/3266478835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ec15f300 0x7f62ec1639f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0097a0 tx=0x7f62dc02a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2419a6318fb143f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:32.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.789+0000 7f62eb7fe640 1 -- 172.21.15.60:0/3266478835 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc037070 con 0x7f62ec15f300 2024-05-09T05:00:32.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.789+0000 7f62eb7fe640 1 -- 172.21.15.60:0/3266478835 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62dc032440 con 0x7f62ec15f300 2024-05-09T05:00:32.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.789+0000 7f62eb7fe640 1 -- 172.21.15.60:0/3266478835 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc032810 con 0x7f62ec15f300 2024-05-09T05:00:32.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.789+0000 7f62f2c99640 1 -- 172.21.15.60:0/3266478835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ec15f300 msgr2=0x7f62ec1639f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:32.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.789+0000 7f62f2c99640 1 --2- 172.21.15.60:0/3266478835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ec15f300 0x7f62ec1639f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0097a0 tx=0x7f62dc02a230 comp rx=0 tx=0).stop 2024-05-09T05:00:32.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.789+0000 7f62f2c99640 1 -- 172.21.15.60:0/3266478835 shutdown_connections 2024-05-09T05:00:32.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.789+0000 7f62f2c99640 1 --2- 172.21.15.60:0/3266478835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ec15f300 0x7f62ec1639f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.789+0000 7f62f2c99640 1 -- 172.21.15.60:0/3266478835 >> 172.21.15.60:0/3266478835 conn(0x7f62ec09b850 msgr2=0x7f62ec09bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:32.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.789+0000 7f62f2c99640 1 -- 172.21.15.60:0/3266478835 shutdown_connections 2024-05-09T05:00:32.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.789+0000 7f62f2c99640 1 -- 172.21.15.60:0/3266478835 wait complete. 2024-05-09T05:00:32.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.791+0000 7f62f2c99640 1 Processor -- start 2024-05-09T05:00:32.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.791+0000 7f62f2c99640 1 -- start start 2024-05-09T05:00:32.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62f2c99640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62ec06f9f0 0x7f62ec09f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62f2c99640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ec15f300 0x7f62ec0ac990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62f2c99640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ec1606f0 0x7f62ec0b29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62f2c99640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec0d3b00 con 0x7f62ec1606f0 2024-05-09T05:00:32.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62f2c99640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec163f30 con 0x7f62ec15f300 2024-05-09T05:00:32.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62f2c99640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f62ec0d3c80 con 0x7f62ec06f9f0 2024-05-09T05:00:32.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62f1236640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ec1606f0 0x7f62ec0b29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:32.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62f1236640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ec1606f0 0x7f62ec0b29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34758/0 (socket says 172.21.15.60:34758) 2024-05-09T05:00:32.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62f1236640 1 -- 172.21.15.60:0/112144042 learned_addr learned my addr 172.21.15.60:0/112144042 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:32.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62f0a35640 1 --2- 172.21.15.60:0/112144042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62ec06f9f0 0x7f62ec09f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:32.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62ebfff640 1 --2- 172.21.15.60:0/112144042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ec15f300 0x7f62ec0ac990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:32.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62f1236640 1 -- 172.21.15.60:0/112144042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62ec06f9f0 msgr2=0x7f62ec09f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:32.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62f1236640 1 --2- 172.21.15.60:0/112144042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62ec06f9f0 0x7f62ec09f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62f1236640 1 -- 172.21.15.60:0/112144042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ec15f300 msgr2=0x7f62ec0ac990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:32.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62f1236640 1 --2- 172.21.15.60:0/112144042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ec15f300 0x7f62ec0ac990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:32.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62f1236640 1 -- 172.21.15.60:0/112144042 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec0aced0 con 0x7f62ec1606f0 2024-05-09T05:00:32.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62f1236640 1 --2- 172.21.15.60:0/112144042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ec1606f0 0x7f62ec0b29e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f62e400c3f0 tx=0x7f62e400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:32.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62e97fa640 1 -- 172.21.15.60:0/112144042 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e4017070 con 0x7f62ec1606f0 2024-05-09T05:00:32.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62e97fa640 1 -- 172.21.15.60:0/112144042 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62e40052d0 con 0x7f62ec1606f0 2024-05-09T05:00:32.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62f2c99640 1 -- 172.21.15.60:0/112144042 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ec0b17f0 con 0x7f62ec1606f0 2024-05-09T05:00:32.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.792+0000 7f62e97fa640 1 -- 172.21.15.60:0/112144042 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e4003810 con 0x7f62ec1606f0 2024-05-09T05:00:32.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.793+0000 7f62f2c99640 1 -- 172.21.15.60:0/112144042 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ec0b1db0 con 0x7f62ec1606f0 2024-05-09T05:00:32.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.793+0000 7f62f2c99640 1 -- 172.21.15.60:0/112144042 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62ec051040 con 0x7f62ec1606f0 2024-05-09T05:00:32.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.794+0000 7f62e97fa640 1 -- 172.21.15.60:0/112144042 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f62e401a070 con 0x7f62ec1606f0 2024-05-09T05:00:32.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.795+0000 7f62e97fa640 1 --2- 172.21.15.60:0/112144042 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f62cc0762d0 0x7f62cc078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:32.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.795+0000 7f62e97fa640 1 -- 172.21.15.60:0/112144042 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f62e4096c10 con 0x7f62ec1606f0 2024-05-09T05:00:32.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.795+0000 7f62f0a35640 1 --2- 172.21.15.60:0/112144042 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f62cc0762d0 0x7f62cc078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:32.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.795+0000 7f62f0a35640 1 --2- 172.21.15.60:0/112144042 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f62cc0762d0 0x7f62cc078790 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f62dc034000 tx=0x7f62dc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:32.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.799+0000 7f62e97fa640 1 -- 172.21.15.60:0/112144042 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f62e40602c0 con 0x7f62ec1606f0 2024-05-09T05:00:32.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:32.991+0000 7f62f2c99640 1 -- 172.21.15.60:0/112144042 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f62ec09f8e0 con 0x7f62cc0762d0 2024-05-09T05:00:33.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.000+0000 7f62e97fa640 1 -- 172.21.15.60:0/112144042 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332523 (secure 0 0 0) 0x7f62ec09f8e0 con 0x7f62cc0762d0 2024-05-09T05:00:33.005 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:33.007 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-05-09T05:00:33.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.004+0000 7f62f2c99640 1 -- 172.21.15.60:0/112144042 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f62cc0762d0 msgr2=0x7f62cc078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.004+0000 7f62f2c99640 1 --2- 172.21.15.60:0/112144042 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f62cc0762d0 0x7f62cc078790 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f62dc034000 tx=0x7f62dc02e000 comp rx=0 tx=0).stop 2024-05-09T05:00:33.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.004+0000 7f62f2c99640 1 -- 172.21.15.60:0/112144042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ec1606f0 msgr2=0x7f62ec0b29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.004+0000 7f62f2c99640 1 --2- 172.21.15.60:0/112144042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ec1606f0 0x7f62ec0b29e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f62e400c3f0 tx=0x7f62e400c8b0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.004+0000 7f62f2c99640 1 -- 172.21.15.60:0/112144042 shutdown_connections 2024-05-09T05:00:33.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.004+0000 7f62f2c99640 1 --2- 172.21.15.60:0/112144042 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f62cc0762d0 0x7f62cc078790 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.004+0000 7f62f2c99640 1 --2- 172.21.15.60:0/112144042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62ec1606f0 0x7f62ec0b29e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.004+0000 7f62f2c99640 1 --2- 172.21.15.60:0/112144042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ec15f300 0x7f62ec0ac990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.004+0000 7f62f2c99640 1 --2- 172.21.15.60:0/112144042 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62ec06f9f0 0x7f62ec09f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.004+0000 7f62f2c99640 1 -- 172.21.15.60:0/112144042 >> 172.21.15.60:0/112144042 conn(0x7f62ec09b850 msgr2=0x7f62ec073930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:33.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.004+0000 7f62f2c99640 1 -- 172.21.15.60:0/112144042 shutdown_connections 2024-05-09T05:00:33.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.004+0000 7f62f2c99640 1 -- 172.21.15.60:0/112144042 wait complete. 2024-05-09T05:00:33.016 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-05-09T05:00:32.915129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223524,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751196,"statfs":{"total":773094113280,"available":772865224704,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003187"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624117+0000","last_change":"2024-05-09T05:00:11.849304+0000","last_active":"2024-05-09T05:00:19.624117+0000","last_peered":"2024-05-09T05:00:19.624117+0000","last_clean":"2024-05-09T05:00:19.624117+0000","last_became_active":"2024-05-09T05:00:11.848977+0000","last_became_peered":"2024-05-09T05:00:11.848977+0000","last_unstale":"2024-05-09T05:00:19.624117+0000","last_undegraded":"2024-05-09T05:00:19.624117+0000","last_fullsized":"2024-05-09T05:00:19.624117+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:48:26.640184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652587+0000","last_change":"2024-05-09T05:00:12.853461+0000","last_active":"2024-05-09T05:00:18.652587+0000","last_peered":"2024-05-09T05:00:18.652587+0000","last_clean":"2024-05-09T05:00:18.652587+0000","last_became_active":"2024-05-09T05:00:12.853171+0000","last_became_peered":"2024-05-09T05:00:12.853171+0000","last_unstale":"2024-05-09T05:00:18.652587+0000","last_undegraded":"2024-05-09T05:00:18.652587+0000","last_fullsized":"2024-05-09T05:00:18.652587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:01:08.488591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652158+0000","last_change":"2024-05-09T05:00:16.883712+0000","last_active":"2024-05-09T05:00:18.652158+0000","last_peered":"2024-05-09T05:00:18.652158+0000","last_clean":"2024-05-09T05:00:18.652158+0000","last_became_active":"2024-05-09T05:00:16.883004+0000","last_became_peered":"2024-05-09T05:00:16.883004+0000","last_unstale":"2024-05-09T05:00:18.652158+0000","last_undegraded":"2024-05-09T05:00:18.652158+0000","last_fullsized":"2024-05-09T05:00:18.652158+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:27.371200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652845+0000","last_change":"2024-05-09T05:00:11.839791+0000","last_active":"2024-05-09T05:00:18.652845+0000","last_peered":"2024-05-09T05:00:18.652845+0000","last_clean":"2024-05-09T05:00:18.652845+0000","last_became_active":"2024-05-09T05:00:11.839531+0000","last_became_peered":"2024-05-09T05:00:11.839531+0000","last_unstale":"2024-05-09T05:00:18.652845+0000","last_undegraded":"2024-05-09T05:00:18.652845+0000","last_fullsized":"2024-05-09T05:00:18.652845+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:23.679256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623978+0000","last_change":"2024-05-09T05:00:12.844866+0000","last_active":"2024-05-09T05:00:19.623978+0000","last_peered":"2024-05-09T05:00:19.623978+0000","last_clean":"2024-05-09T05:00:19.623978+0000","last_became_active":"2024-05-09T05:00:12.844015+0000","last_became_peered":"2024-05-09T05:00:12.844015+0000","last_unstale":"2024-05-09T05:00:19.623978+0000","last_undegraded":"2024-05-09T05:00:19.623978+0000","last_fullsized":"2024-05-09T05:00:19.623978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881734+0000","last_change":"2024-05-09T05:00:16.878021+0000","last_active":"2024-05-09T05:00:19.881734+0000","last_peered":"2024-05-09T05:00:19.881734+0000","last_clean":"2024-05-09T05:00:19.881734+0000","last_became_active":"2024-05-09T05:00:16.877322+0000","last_became_peered":"2024-05-09T05:00:16.877322+0000","last_unstale":"2024-05-09T05:00:19.881734+0000","last_undegraded":"2024-05-09T05:00:19.881734+0000","last_fullsized":"2024-05-09T05:00:19.881734+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:34.719408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270583+0000","last_change":"2024-05-09T05:00:11.852994+0000","last_active":"2024-05-09T05:00:20.270583+0000","last_peered":"2024-05-09T05:00:20.270583+0000","last_clean":"2024-05-09T05:00:20.270583+0000","last_became_active":"2024-05-09T05:00:11.852737+0000","last_became_peered":"2024-05-09T05:00:11.852737+0000","last_unstale":"2024-05-09T05:00:20.270583+0000","last_undegraded":"2024-05-09T05:00:20.270583+0000","last_fullsized":"2024-05-09T05:00:20.270583+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:39.794811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970762+0000","last_change":"2024-05-09T05:00:12.845198+0000","last_active":"2024-05-09T05:00:19.970762+0000","last_peered":"2024-05-09T05:00:19.970762+0000","last_clean":"2024-05-09T05:00:19.970762+0000","last_became_active":"2024-05-09T05:00:12.844003+0000","last_became_peered":"2024-05-09T05:00:12.844003+0000","last_unstale":"2024-05-09T05:00:19.970762+0000","last_undegraded":"2024-05-09T05:00:19.970762+0000","last_fullsized":"2024-05-09T05:00:19.970762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:45.314128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882179+0000","last_change":"2024-05-09T05:00:16.877475+0000","last_active":"2024-05-09T05:00:19.882179+0000","last_peered":"2024-05-09T05:00:19.882179+0000","last_clean":"2024-05-09T05:00:19.882179+0000","last_became_active":"2024-05-09T05:00:16.876947+0000","last_became_peered":"2024-05-09T05:00:16.876947+0000","last_unstale":"2024-05-09T05:00:19.882179+0000","last_undegraded":"2024-05-09T05:00:19.882179+0000","last_fullsized":"2024-05-09T05:00:19.882179+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:49.342204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271162+0000","last_change":"2024-05-09T05:00:11.853073+0000","last_active":"2024-05-09T05:00:20.271162+0000","last_peered":"2024-05-09T05:00:20.271162+0000","last_clean":"2024-05-09T05:00:20.271162+0000","last_became_active":"2024-05-09T05:00:11.852759+0000","last_became_peered":"2024-05-09T05:00:11.852759+0000","last_unstale":"2024-05-09T05:00:20.271162+0000","last_undegraded":"2024-05-09T05:00:20.271162+0000","last_fullsized":"2024-05-09T05:00:20.271162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:01.738771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970943+0000","last_change":"2024-05-09T05:00:12.844419+0000","last_active":"2024-05-09T05:00:19.970943+0000","last_peered":"2024-05-09T05:00:19.970943+0000","last_clean":"2024-05-09T05:00:19.970943+0000","last_became_active":"2024-05-09T05:00:12.843951+0000","last_became_peered":"2024-05-09T05:00:12.843951+0000","last_unstale":"2024-05-09T05:00:19.970943+0000","last_undegraded":"2024-05-09T05:00:19.970943+0000","last_fullsized":"2024-05-09T05:00:19.970943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:23.066586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881928+0000","last_change":"2024-05-09T05:00:16.875575+0000","last_active":"2024-05-09T05:00:19.881928+0000","last_peered":"2024-05-09T05:00:19.881928+0000","last_clean":"2024-05-09T05:00:19.881928+0000","last_became_active":"2024-05-09T05:00:16.874993+0000","last_became_peered":"2024-05-09T05:00:16.874993+0000","last_unstale":"2024-05-09T05:00:19.881928+0000","last_undegraded":"2024-05-09T05:00:19.881928+0000","last_fullsized":"2024-05-09T05:00:19.881928+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:34.414100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860116+0000","last_change":"2024-05-09T05:00:12.844310+0000","last_active":"2024-05-09T05:00:17.860116+0000","last_peered":"2024-05-09T05:00:17.860116+0000","last_clean":"2024-05-09T05:00:17.860116+0000","last_became_active":"2024-05-09T05:00:12.843870+0000","last_became_peered":"2024-05-09T05:00:12.843870+0000","last_unstale":"2024-05-09T05:00:17.860116+0000","last_undegraded":"2024-05-09T05:00:17.860116+0000","last_fullsized":"2024-05-09T05:00:17.860116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:19.080163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271207+0000","last_change":"2024-05-09T05:00:11.849654+0000","last_active":"2024-05-09T05:00:20.271207+0000","last_peered":"2024-05-09T05:00:20.271207+0000","last_clean":"2024-05-09T05:00:20.271207+0000","last_became_active":"2024-05-09T05:00:11.849112+0000","last_became_peered":"2024-05-09T05:00:11.849112+0000","last_unstale":"2024-05-09T05:00:20.271207+0000","last_undegraded":"2024-05-09T05:00:20.271207+0000","last_fullsized":"2024-05-09T05:00:20.271207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:04:53.801780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882282+0000","last_change":"2024-05-09T05:00:16.874883+0000","last_active":"2024-05-09T05:00:19.882282+0000","last_peered":"2024-05-09T05:00:19.882282+0000","last_clean":"2024-05-09T05:00:19.882282+0000","last_became_active":"2024-05-09T05:00:16.874390+0000","last_became_peered":"2024-05-09T05:00:16.874390+0000","last_unstale":"2024-05-09T05:00:19.882282+0000","last_undegraded":"2024-05-09T05:00:19.882282+0000","last_fullsized":"2024-05-09T05:00:19.882282+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:09.005203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652613+0000","last_change":"2024-05-09T05:00:12.855440+0000","last_active":"2024-05-09T05:00:18.652613+0000","last_peered":"2024-05-09T05:00:18.652613+0000","last_clean":"2024-05-09T05:00:18.652613+0000","last_became_active":"2024-05-09T05:00:12.854274+0000","last_became_peered":"2024-05-09T05:00:12.854274+0000","last_unstale":"2024-05-09T05:00:18.652613+0000","last_undegraded":"2024-05-09T05:00:18.652613+0000","last_fullsized":"2024-05-09T05:00:18.652613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:30.691332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270354+0000","last_change":"2024-05-09T05:00:11.842122+0000","last_active":"2024-05-09T05:00:20.270354+0000","last_peered":"2024-05-09T05:00:20.270354+0000","last_clean":"2024-05-09T05:00:20.270354+0000","last_became_active":"2024-05-09T05:00:11.840835+0000","last_became_peered":"2024-05-09T05:00:11.840835+0000","last_unstale":"2024-05-09T05:00:20.270354+0000","last_undegraded":"2024-05-09T05:00:20.270354+0000","last_fullsized":"2024-05-09T05:00:20.270354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:41:27.364045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882288+0000","last_change":"2024-05-09T05:00:16.875354+0000","last_active":"2024-05-09T05:00:19.882288+0000","last_peered":"2024-05-09T05:00:19.882288+0000","last_clean":"2024-05-09T05:00:19.882288+0000","last_became_active":"2024-05-09T05:00:16.874942+0000","last_became_peered":"2024-05-09T05:00:16.874942+0000","last_unstale":"2024-05-09T05:00:19.882288+0000","last_undegraded":"2024-05-09T05:00:19.882288+0000","last_fullsized":"2024-05-09T05:00:19.882288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:27.094662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652325+0000","last_change":"2024-05-09T05:00:12.853552+0000","last_active":"2024-05-09T05:00:18.652325+0000","last_peered":"2024-05-09T05:00:18.652325+0000","last_clean":"2024-05-09T05:00:18.652325+0000","last_became_active":"2024-05-09T05:00:12.853268+0000","last_became_peered":"2024-05-09T05:00:12.853268+0000","last_unstale":"2024-05-09T05:00:18.652325+0000","last_undegraded":"2024-05-09T05:00:18.652325+0000","last_fullsized":"2024-05-09T05:00:18.652325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:44.359533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270643+0000","last_change":"2024-05-09T05:00:11.848638+0000","last_active":"2024-05-09T05:00:20.270643+0000","last_peered":"2024-05-09T05:00:20.270643+0000","last_clean":"2024-05-09T05:00:20.270643+0000","last_became_active":"2024-05-09T05:00:11.848351+0000","last_became_peered":"2024-05-09T05:00:11.848351+0000","last_unstale":"2024-05-09T05:00:20.270643+0000","last_undegraded":"2024-05-09T05:00:20.270643+0000","last_fullsized":"2024-05-09T05:00:20.270643+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:01.738771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271321+0000","last_change":"2024-05-09T05:00:16.880744+0000","last_active":"2024-05-09T05:00:20.271321+0000","last_peered":"2024-05-09T05:00:20.271321+0000","last_clean":"2024-05-09T05:00:20.271321+0000","last_became_active":"2024-05-09T05:00:16.880373+0000","last_became_peered":"2024-05-09T05:00:16.880373+0000","last_unstale":"2024-05-09T05:00:20.271321+0000","last_undegraded":"2024-05-09T05:00:20.271321+0000","last_fullsized":"2024-05-09T05:00:20.271321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:10.821888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882019+0000","last_change":"2024-05-09T05:00:12.835034+0000","last_active":"2024-05-09T05:00:19.882019+0000","last_peered":"2024-05-09T05:00:19.882019+0000","last_clean":"2024-05-09T05:00:19.882019+0000","last_became_active":"2024-05-09T05:00:12.834744+0000","last_became_peered":"2024-05-09T05:00:12.834744+0000","last_unstale":"2024-05-09T05:00:19.882019+0000","last_undegraded":"2024-05-09T05:00:19.882019+0000","last_fullsized":"2024-05-09T05:00:19.882019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:26.805203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270527+0000","last_change":"2024-05-09T05:00:11.848682+0000","last_active":"2024-05-09T05:00:20.270527+0000","last_peered":"2024-05-09T05:00:20.270527+0000","last_clean":"2024-05-09T05:00:20.270527+0000","last_became_active":"2024-05-09T05:00:11.848396+0000","last_became_peered":"2024-05-09T05:00:11.848396+0000","last_unstale":"2024-05-09T05:00:20.270527+0000","last_undegraded":"2024-05-09T05:00:20.270527+0000","last_fullsized":"2024-05-09T05:00:20.270527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:44:46.364938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882265+0000","last_change":"2024-05-09T05:00:16.877861+0000","last_active":"2024-05-09T05:00:19.882265+0000","last_peered":"2024-05-09T05:00:19.882265+0000","last_clean":"2024-05-09T05:00:19.882265+0000","last_became_active":"2024-05-09T05:00:16.877155+0000","last_became_peered":"2024-05-09T05:00:16.877155+0000","last_unstale":"2024-05-09T05:00:19.882265+0000","last_undegraded":"2024-05-09T05:00:19.882265+0000","last_fullsized":"2024-05-09T05:00:19.882265+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:01:12.516667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624129+0000","last_change":"2024-05-09T05:00:12.852207+0000","last_active":"2024-05-09T05:00:19.624129+0000","last_peered":"2024-05-09T05:00:19.624129+0000","last_clean":"2024-05-09T05:00:19.624129+0000","last_became_active":"2024-05-09T05:00:12.851927+0000","last_became_peered":"2024-05-09T05:00:12.851927+0000","last_unstale":"2024-05-09T05:00:19.624129+0000","last_undegraded":"2024-05-09T05:00:19.624129+0000","last_fullsized":"2024-05-09T05:00:19.624129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:44.359533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.861099+0000","last_change":"2024-05-09T05:00:11.849008+0000","last_active":"2024-05-09T05:00:17.861099+0000","last_peered":"2024-05-09T05:00:17.861099+0000","last_clean":"2024-05-09T05:00:17.861099+0000","last_became_active":"2024-05-09T05:00:11.848741+0000","last_became_peered":"2024-05-09T05:00:11.848741+0000","last_unstale":"2024-05-09T05:00:17.861099+0000","last_undegraded":"2024-05-09T05:00:17.861099+0000","last_fullsized":"2024-05-09T05:00:17.861099+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:39.794811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881982+0000","last_change":"2024-05-09T05:00:16.879683+0000","last_active":"2024-05-09T05:00:19.881982+0000","last_peered":"2024-05-09T05:00:19.881982+0000","last_clean":"2024-05-09T05:00:19.881982+0000","last_became_active":"2024-05-09T05:00:16.879337+0000","last_became_peered":"2024-05-09T05:00:16.879337+0000","last_unstale":"2024-05-09T05:00:19.881982+0000","last_undegraded":"2024-05-09T05:00:19.881982+0000","last_fullsized":"2024-05-09T05:00:19.881982+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:48.387609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652854+0000","last_change":"2024-05-09T05:00:12.852575+0000","last_active":"2024-05-09T05:00:18.652854+0000","last_peered":"2024-05-09T05:00:18.652854+0000","last_clean":"2024-05-09T05:00:18.652854+0000","last_became_active":"2024-05-09T05:00:12.852308+0000","last_became_peered":"2024-05-09T05:00:12.852308+0000","last_unstale":"2024-05-09T05:00:18.652854+0000","last_undegraded":"2024-05-09T05:00:18.652854+0000","last_fullsized":"2024-05-09T05:00:18.652854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270373+0000","last_change":"2024-05-09T05:00:11.839637+0000","last_active":"2024-05-09T05:00:20.270373+0000","last_peered":"2024-05-09T05:00:20.270373+0000","last_clean":"2024-05-09T05:00:20.270373+0000","last_became_active":"2024-05-09T05:00:11.839214+0000","last_became_peered":"2024-05-09T05:00:11.839214+0000","last_unstale":"2024-05-09T05:00:20.270373+0000","last_undegraded":"2024-05-09T05:00:20.270373+0000","last_fullsized":"2024-05-09T05:00:20.270373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:25.803010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652033+0000","last_change":"2024-05-09T05:00:16.883218+0000","last_active":"2024-05-09T05:00:18.652033+0000","last_peered":"2024-05-09T05:00:18.652033+0000","last_clean":"2024-05-09T05:00:18.652033+0000","last_became_active":"2024-05-09T05:00:16.882353+0000","last_became_peered":"2024-05-09T05:00:16.882353+0000","last_unstale":"2024-05-09T05:00:18.652033+0000","last_undegraded":"2024-05-09T05:00:18.652033+0000","last_fullsized":"2024-05-09T05:00:18.652033+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:02.171370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271136+0000","last_change":"2024-05-09T05:00:12.846656+0000","last_active":"2024-05-09T05:00:20.271136+0000","last_peered":"2024-05-09T05:00:20.271136+0000","last_clean":"2024-05-09T05:00:20.271136+0000","last_became_active":"2024-05-09T05:00:12.846359+0000","last_became_peered":"2024-05-09T05:00:12.846359+0000","last_unstale":"2024-05-09T05:00:20.271136+0000","last_undegraded":"2024-05-09T05:00:20.271136+0000","last_fullsized":"2024-05-09T05:00:20.271136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:49.330395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270574+0000","last_change":"2024-05-09T05:00:11.850170+0000","last_active":"2024-05-09T05:00:20.270574+0000","last_peered":"2024-05-09T05:00:20.270574+0000","last_clean":"2024-05-09T05:00:20.270574+0000","last_became_active":"2024-05-09T05:00:11.849866+0000","last_became_peered":"2024-05-09T05:00:11.849866+0000","last_unstale":"2024-05-09T05:00:20.270574+0000","last_undegraded":"2024-05-09T05:00:20.270574+0000","last_fullsized":"2024-05-09T05:00:20.270574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:27:57.420062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623671+0000","last_change":"2024-05-09T05:00:16.879852+0000","last_active":"2024-05-09T05:00:19.623671+0000","last_peered":"2024-05-09T05:00:19.623671+0000","last_clean":"2024-05-09T05:00:19.623671+0000","last_became_active":"2024-05-09T05:00:16.878219+0000","last_became_peered":"2024-05-09T05:00:16.878219+0000","last_unstale":"2024-05-09T05:00:19.623671+0000","last_undegraded":"2024-05-09T05:00:19.623671+0000","last_fullsized":"2024-05-09T05:00:19.623671+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:20.415639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.971002+0000","last_change":"2024-05-09T05:00:12.859874+0000","last_active":"2024-05-09T05:00:19.971002+0000","last_peered":"2024-05-09T05:00:19.971002+0000","last_clean":"2024-05-09T05:00:19.971002+0000","last_became_active":"2024-05-09T05:00:12.859620+0000","last_became_peered":"2024-05-09T05:00:12.859620+0000","last_unstale":"2024-05-09T05:00:19.971002+0000","last_undegraded":"2024-05-09T05:00:19.971002+0000","last_fullsized":"2024-05-09T05:00:19.971002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:04.977127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270358+0000","last_change":"2024-05-09T05:00:11.838593+0000","last_active":"2024-05-09T05:00:20.270358+0000","last_peered":"2024-05-09T05:00:20.270358+0000","last_clean":"2024-05-09T05:00:20.270358+0000","last_became_active":"2024-05-09T05:00:11.838314+0000","last_became_peered":"2024-05-09T05:00:11.838314+0000","last_unstale":"2024-05-09T05:00:20.270358+0000","last_undegraded":"2024-05-09T05:00:20.270358+0000","last_fullsized":"2024-05-09T05:00:20.270358+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:49.660089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270704+0000","last_change":"2024-05-09T05:00:16.876934+0000","last_active":"2024-05-09T05:00:20.270704+0000","last_peered":"2024-05-09T05:00:20.270704+0000","last_clean":"2024-05-09T05:00:20.270704+0000","last_became_active":"2024-05-09T05:00:16.876531+0000","last_became_peered":"2024-05-09T05:00:16.876531+0000","last_unstale":"2024-05-09T05:00:20.270704+0000","last_undegraded":"2024-05-09T05:00:20.270704+0000","last_fullsized":"2024-05-09T05:00:20.270704+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:54.996456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882435+0000","last_change":"2024-05-09T05:00:12.849957+0000","last_active":"2024-05-09T05:00:19.882435+0000","last_peered":"2024-05-09T05:00:19.882435+0000","last_clean":"2024-05-09T05:00:19.882435+0000","last_became_active":"2024-05-09T05:00:12.849138+0000","last_became_peered":"2024-05-09T05:00:12.849138+0000","last_unstale":"2024-05-09T05:00:19.882435+0000","last_undegraded":"2024-05-09T05:00:19.882435+0000","last_fullsized":"2024-05-09T05:00:19.882435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:02.740965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970665+0000","last_change":"2024-05-09T05:00:11.840734+0000","last_active":"2024-05-09T05:00:19.970665+0000","last_peered":"2024-05-09T05:00:19.970665+0000","last_clean":"2024-05-09T05:00:19.970665+0000","last_became_active":"2024-05-09T05:00:11.840463+0000","last_became_peered":"2024-05-09T05:00:11.840463+0000","last_unstale":"2024-05-09T05:00:19.970665+0000","last_undegraded":"2024-05-09T05:00:19.970665+0000","last_fullsized":"2024-05-09T05:00:19.970665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:49.660089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271060+0000","last_change":"2024-05-09T05:00:16.875120+0000","last_active":"2024-05-09T05:00:20.271060+0000","last_peered":"2024-05-09T05:00:20.271060+0000","last_clean":"2024-05-09T05:00:20.271060+0000","last_became_active":"2024-05-09T05:00:16.874728+0000","last_became_peered":"2024-05-09T05:00:16.874728+0000","last_unstale":"2024-05-09T05:00:20.271060+0000","last_undegraded":"2024-05-09T05:00:20.271060+0000","last_fullsized":"2024-05-09T05:00:20.271060+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:20.415639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271161+0000","last_change":"2024-05-09T05:00:12.851489+0000","last_active":"2024-05-09T05:00:20.271161+0000","last_peered":"2024-05-09T05:00:20.271161+0000","last_clean":"2024-05-09T05:00:20.271161+0000","last_became_active":"2024-05-09T05:00:12.841469+0000","last_became_peered":"2024-05-09T05:00:12.841469+0000","last_unstale":"2024-05-09T05:00:20.271161+0000","last_undegraded":"2024-05-09T05:00:20.271161+0000","last_fullsized":"2024-05-09T05:00:20.271161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:45.463031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860713+0000","last_change":"2024-05-09T05:00:11.841055+0000","last_active":"2024-05-09T05:00:17.860713+0000","last_peered":"2024-05-09T05:00:17.860713+0000","last_clean":"2024-05-09T05:00:17.860713+0000","last_became_active":"2024-05-09T05:00:11.839965+0000","last_became_peered":"2024-05-09T05:00:11.839965+0000","last_unstale":"2024-05-09T05:00:17.860713+0000","last_undegraded":"2024-05-09T05:00:17.860713+0000","last_fullsized":"2024-05-09T05:00:17.860713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:41:27.364045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271386+0000","last_change":"2024-05-09T05:00:16.873524+0000","last_active":"2024-05-09T05:00:20.271386+0000","last_peered":"2024-05-09T05:00:20.271386+0000","last_clean":"2024-05-09T05:00:20.271386+0000","last_became_active":"2024-05-09T05:00:16.872894+0000","last_became_peered":"2024-05-09T05:00:16.872894+0000","last_unstale":"2024-05-09T05:00:20.271386+0000","last_undegraded":"2024-05-09T05:00:20.271386+0000","last_fullsized":"2024-05-09T05:00:20.271386+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:32:00.425543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623506+0000","last_change":"2024-05-09T05:00:12.849280+0000","last_active":"2024-05-09T05:00:19.623506+0000","last_peered":"2024-05-09T05:00:19.623506+0000","last_clean":"2024-05-09T05:00:19.623506+0000","last_became_active":"2024-05-09T05:00:12.848968+0000","last_became_peered":"2024-05-09T05:00:12.848968+0000","last_unstale":"2024-05-09T05:00:19.623506+0000","last_undegraded":"2024-05-09T05:00:19.623506+0000","last_fullsized":"2024-05-09T05:00:19.623506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:49.330395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270849+0000","last_change":"2024-05-09T05:00:11.852903+0000","last_active":"2024-05-09T05:00:20.270849+0000","last_peered":"2024-05-09T05:00:20.270849+0000","last_clean":"2024-05-09T05:00:20.270849+0000","last_became_active":"2024-05-09T05:00:11.852584+0000","last_became_peered":"2024-05-09T05:00:11.852584+0000","last_unstale":"2024-05-09T05:00:20.270849+0000","last_undegraded":"2024-05-09T05:00:20.270849+0000","last_fullsized":"2024-05-09T05:00:20.270849+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:29:13.766278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624019+0000","last_change":"2024-05-09T05:00:16.882378+0000","last_active":"2024-05-09T05:00:19.624019+0000","last_peered":"2024-05-09T05:00:19.624019+0000","last_clean":"2024-05-09T05:00:19.624019+0000","last_became_active":"2024-05-09T05:00:16.881947+0000","last_became_peered":"2024-05-09T05:00:16.881947+0000","last_unstale":"2024-05-09T05:00:19.624019+0000","last_undegraded":"2024-05-09T05:00:19.624019+0000","last_fullsized":"2024-05-09T05:00:19.624019+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:32:00.425543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652370+0000","last_change":"2024-05-09T05:00:12.847632+0000","last_active":"2024-05-09T05:00:18.652370+0000","last_peered":"2024-05-09T05:00:18.652370+0000","last_clean":"2024-05-09T05:00:18.652370+0000","last_became_active":"2024-05-09T05:00:12.847358+0000","last_became_peered":"2024-05-09T05:00:12.847358+0000","last_unstale":"2024-05-09T05:00:18.652370+0000","last_undegraded":"2024-05-09T05:00:18.652370+0000","last_fullsized":"2024-05-09T05:00:18.652370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:49.330395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970011+0000","last_change":"2024-05-09T05:00:11.842747+0000","last_active":"2024-05-09T05:00:19.970011+0000","last_peered":"2024-05-09T05:00:19.970011+0000","last_clean":"2024-05-09T05:00:19.970011+0000","last_became_active":"2024-05-09T05:00:11.842474+0000","last_became_peered":"2024-05-09T05:00:11.842474+0000","last_unstale":"2024-05-09T05:00:19.970011+0000","last_undegraded":"2024-05-09T05:00:19.970011+0000","last_fullsized":"2024-05-09T05:00:19.970011+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:01.738771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":118,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271122+0000","last_change":"2024-05-09T05:00:08.667158+0000","last_active":"2024-05-09T05:00:20.271122+0000","last_peered":"2024-05-09T05:00:20.271122+0000","last_clean":"2024-05-09T05:00:20.271122+0000","last_became_active":"2024-05-09T05:00:08.666316+0000","last_became_peered":"2024-05-09T05:00:08.666316+0000","last_unstale":"2024-05-09T05:00:20.271122+0000","last_undegraded":"2024-05-09T05:00:20.271122+0000","last_fullsized":"2024-05-09T05:00:20.271122+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_clean_scrub_stamp":"2024-05-09T05:00:07.370119+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:20.234643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860739+0000","last_change":"2024-05-09T05:00:16.880367+0000","last_active":"2024-05-09T05:00:17.860739+0000","last_peered":"2024-05-09T05:00:17.860739+0000","last_clean":"2024-05-09T05:00:17.860739+0000","last_became_active":"2024-05-09T05:00:16.879980+0000","last_became_peered":"2024-05-09T05:00:16.879980+0000","last_unstale":"2024-05-09T05:00:17.860739+0000","last_undegraded":"2024-05-09T05:00:17.860739+0000","last_fullsized":"2024-05-09T05:00:17.860739+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:54.996456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271273+0000","last_change":"2024-05-09T05:00:12.852180+0000","last_active":"2024-05-09T05:00:20.271273+0000","last_peered":"2024-05-09T05:00:20.271273+0000","last_clean":"2024-05-09T05:00:20.271273+0000","last_became_active":"2024-05-09T05:00:12.851828+0000","last_became_peered":"2024-05-09T05:00:12.851828+0000","last_unstale":"2024-05-09T05:00:20.271273+0000","last_undegraded":"2024-05-09T05:00:20.271273+0000","last_fullsized":"2024-05-09T05:00:20.271273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:16.387563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.861065+0000","last_change":"2024-05-09T05:00:11.843024+0000","last_active":"2024-05-09T05:00:17.861065+0000","last_peered":"2024-05-09T05:00:17.861065+0000","last_clean":"2024-05-09T05:00:17.861065+0000","last_became_active":"2024-05-09T05:00:11.842703+0000","last_became_peered":"2024-05-09T05:00:11.842703+0000","last_unstale":"2024-05-09T05:00:17.861065+0000","last_undegraded":"2024-05-09T05:00:17.861065+0000","last_fullsized":"2024-05-09T05:00:17.861065+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:48:26.640184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860188+0000","last_change":"2024-05-09T05:00:16.879385+0000","last_active":"2024-05-09T05:00:17.860188+0000","last_peered":"2024-05-09T05:00:17.860188+0000","last_clean":"2024-05-09T05:00:17.860188+0000","last_became_active":"2024-05-09T05:00:16.878956+0000","last_became_peered":"2024-05-09T05:00:16.878956+0000","last_unstale":"2024-05-09T05:00:17.860188+0000","last_undegraded":"2024-05-09T05:00:17.860188+0000","last_fullsized":"2024-05-09T05:00:17.860188+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:49.342204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970802+0000","last_change":"2024-05-09T05:00:12.848110+0000","last_active":"2024-05-09T05:00:19.970802+0000","last_peered":"2024-05-09T05:00:19.970802+0000","last_clean":"2024-05-09T05:00:19.970802+0000","last_became_active":"2024-05-09T05:00:12.847628+0000","last_became_peered":"2024-05-09T05:00:12.847628+0000","last_unstale":"2024-05-09T05:00:19.970802+0000","last_undegraded":"2024-05-09T05:00:19.970802+0000","last_fullsized":"2024-05-09T05:00:19.970802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:16.387563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271019+0000","last_change":"2024-05-09T05:00:11.845441+0000","last_active":"2024-05-09T05:00:20.271019+0000","last_peered":"2024-05-09T05:00:20.271019+0000","last_clean":"2024-05-09T05:00:20.271019+0000","last_became_active":"2024-05-09T05:00:11.845151+0000","last_became_peered":"2024-05-09T05:00:11.845151+0000","last_unstale":"2024-05-09T05:00:20.271019+0000","last_undegraded":"2024-05-09T05:00:20.271019+0000","last_fullsized":"2024-05-09T05:00:20.271019+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:39.794811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623840+0000","last_change":"2024-05-09T05:00:16.880872+0000","last_active":"2024-05-09T05:00:19.623840+0000","last_peered":"2024-05-09T05:00:19.623840+0000","last_clean":"2024-05-09T05:00:19.623840+0000","last_became_active":"2024-05-09T05:00:16.876853+0000","last_became_peered":"2024-05-09T05:00:16.876853+0000","last_unstale":"2024-05-09T05:00:19.623840+0000","last_undegraded":"2024-05-09T05:00:19.623840+0000","last_fullsized":"2024-05-09T05:00:19.623840+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:47:02.206708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271556+0000","last_change":"2024-05-09T05:00:12.854694+0000","last_active":"2024-05-09T05:00:20.271556+0000","last_peered":"2024-05-09T05:00:20.271556+0000","last_clean":"2024-05-09T05:00:20.271556+0000","last_became_active":"2024-05-09T05:00:12.854098+0000","last_became_peered":"2024-05-09T05:00:12.854098+0000","last_unstale":"2024-05-09T05:00:20.271556+0000","last_undegraded":"2024-05-09T05:00:20.271556+0000","last_fullsized":"2024-05-09T05:00:20.271556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:23.343124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860998+0000","last_change":"2024-05-09T05:00:11.835134+0000","last_active":"2024-05-09T05:00:17.860998+0000","last_peered":"2024-05-09T05:00:17.860998+0000","last_clean":"2024-05-09T05:00:17.860998+0000","last_became_active":"2024-05-09T05:00:11.834773+0000","last_became_peered":"2024-05-09T05:00:11.834773+0000","last_unstale":"2024-05-09T05:00:17.860998+0000","last_undegraded":"2024-05-09T05:00:17.860998+0000","last_fullsized":"2024-05-09T05:00:17.860998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:50.824636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970193+0000","last_change":"2024-05-09T05:00:16.879633+0000","last_active":"2024-05-09T05:00:19.970193+0000","last_peered":"2024-05-09T05:00:19.970193+0000","last_clean":"2024-05-09T05:00:19.970193+0000","last_became_active":"2024-05-09T05:00:16.879232+0000","last_became_peered":"2024-05-09T05:00:16.879232+0000","last_unstale":"2024-05-09T05:00:19.970193+0000","last_undegraded":"2024-05-09T05:00:19.970193+0000","last_fullsized":"2024-05-09T05:00:19.970193+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:18:15.413800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271558+0000","last_change":"2024-05-09T05:00:12.852679+0000","last_active":"2024-05-09T05:00:20.271558+0000","last_peered":"2024-05-09T05:00:20.271558+0000","last_clean":"2024-05-09T05:00:20.271558+0000","last_became_active":"2024-05-09T05:00:12.852434+0000","last_became_peered":"2024-05-09T05:00:12.852434+0000","last_unstale":"2024-05-09T05:00:20.271558+0000","last_undegraded":"2024-05-09T05:00:20.271558+0000","last_fullsized":"2024-05-09T05:00:20.271558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:06.793812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881482+0000","last_change":"2024-05-09T05:00:11.845447+0000","last_active":"2024-05-09T05:00:19.881482+0000","last_peered":"2024-05-09T05:00:19.881482+0000","last_clean":"2024-05-09T05:00:19.881482+0000","last_became_active":"2024-05-09T05:00:11.845139+0000","last_became_peered":"2024-05-09T05:00:11.845139+0000","last_unstale":"2024-05-09T05:00:19.881482+0000","last_undegraded":"2024-05-09T05:00:19.881482+0000","last_fullsized":"2024-05-09T05:00:19.881482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:32:22.225189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881692+0000","last_change":"2024-05-09T05:00:16.877859+0000","last_active":"2024-05-09T05:00:19.881692+0000","last_peered":"2024-05-09T05:00:19.881692+0000","last_clean":"2024-05-09T05:00:19.881692+0000","last_became_active":"2024-05-09T05:00:16.877418+0000","last_became_peered":"2024-05-09T05:00:16.877418+0000","last_unstale":"2024-05-09T05:00:19.881692+0000","last_undegraded":"2024-05-09T05:00:19.881692+0000","last_fullsized":"2024-05-09T05:00:19.881692+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:28.924332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271116+0000","last_change":"2024-05-09T05:00:12.855313+0000","last_active":"2024-05-09T05:00:20.271116+0000","last_peered":"2024-05-09T05:00:20.271116+0000","last_clean":"2024-05-09T05:00:20.271116+0000","last_became_active":"2024-05-09T05:00:12.855041+0000","last_became_peered":"2024-05-09T05:00:12.855041+0000","last_unstale":"2024-05-09T05:00:20.271116+0000","last_undegraded":"2024-05-09T05:00:20.271116+0000","last_fullsized":"2024-05-09T05:00:20.271116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:01:08.488591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860579+0000","last_change":"2024-05-09T05:00:11.839443+0000","last_active":"2024-05-09T05:00:17.860579+0000","last_peered":"2024-05-09T05:00:17.860579+0000","last_clean":"2024-05-09T05:00:17.860579+0000","last_became_active":"2024-05-09T05:00:11.839169+0000","last_became_peered":"2024-05-09T05:00:11.839169+0000","last_unstale":"2024-05-09T05:00:17.860579+0000","last_undegraded":"2024-05-09T05:00:17.860579+0000","last_fullsized":"2024-05-09T05:00:17.860579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:10.779395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623875+0000","last_change":"2024-05-09T05:00:16.876207+0000","last_active":"2024-05-09T05:00:19.623875+0000","last_peered":"2024-05-09T05:00:19.623875+0000","last_clean":"2024-05-09T05:00:19.623875+0000","last_became_active":"2024-05-09T05:00:16.875739+0000","last_became_peered":"2024-05-09T05:00:16.875739+0000","last_unstale":"2024-05-09T05:00:19.623875+0000","last_undegraded":"2024-05-09T05:00:19.623875+0000","last_fullsized":"2024-05-09T05:00:19.623875+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:10.821888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860230+0000","last_change":"2024-05-09T05:00:12.851872+0000","last_active":"2024-05-09T05:00:17.860230+0000","last_peered":"2024-05-09T05:00:17.860230+0000","last_clean":"2024-05-09T05:00:17.860230+0000","last_became_active":"2024-05-09T05:00:12.851590+0000","last_became_peered":"2024-05-09T05:00:12.851590+0000","last_unstale":"2024-05-09T05:00:17.860230+0000","last_undegraded":"2024-05-09T05:00:17.860230+0000","last_fullsized":"2024-05-09T05:00:17.860230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:49.330395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860330+0000","last_change":"2024-05-09T05:00:11.845978+0000","last_active":"2024-05-09T05:00:17.860330+0000","last_peered":"2024-05-09T05:00:17.860330+0000","last_clean":"2024-05-09T05:00:17.860330+0000","last_became_active":"2024-05-09T05:00:11.845129+0000","last_became_peered":"2024-05-09T05:00:11.845129+0000","last_unstale":"2024-05-09T05:00:17.860330+0000","last_undegraded":"2024-05-09T05:00:17.860330+0000","last_fullsized":"2024-05-09T05:00:17.860330+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:49.660089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652710+0000","last_change":"2024-05-09T05:00:16.882756+0000","last_active":"2024-05-09T05:00:18.652710+0000","last_peered":"2024-05-09T05:00:18.652710+0000","last_clean":"2024-05-09T05:00:18.652710+0000","last_became_active":"2024-05-09T05:00:16.882378+0000","last_became_peered":"2024-05-09T05:00:16.882378+0000","last_unstale":"2024-05-09T05:00:18.652710+0000","last_undegraded":"2024-05-09T05:00:18.652710+0000","last_fullsized":"2024-05-09T05:00:18.652710+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:51:44.393878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271158+0000","last_change":"2024-05-09T05:00:12.854634+0000","last_active":"2024-05-09T05:00:20.271158+0000","last_peered":"2024-05-09T05:00:20.271158+0000","last_clean":"2024-05-09T05:00:20.271158+0000","last_became_active":"2024-05-09T05:00:12.854366+0000","last_became_peered":"2024-05-09T05:00:12.854366+0000","last_unstale":"2024-05-09T05:00:20.271158+0000","last_undegraded":"2024-05-09T05:00:20.271158+0000","last_fullsized":"2024-05-09T05:00:20.271158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:45.314128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.969990+0000","last_change":"2024-05-09T05:00:11.843045+0000","last_active":"2024-05-09T05:00:19.969990+0000","last_peered":"2024-05-09T05:00:19.969990+0000","last_clean":"2024-05-09T05:00:19.969990+0000","last_became_active":"2024-05-09T05:00:11.842728+0000","last_became_peered":"2024-05-09T05:00:11.842728+0000","last_unstale":"2024-05-09T05:00:19.969990+0000","last_undegraded":"2024-05-09T05:00:19.969990+0000","last_fullsized":"2024-05-09T05:00:19.969990+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:25.803010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652628+0000","last_change":"2024-05-09T05:00:16.880672+0000","last_active":"2024-05-09T05:00:18.652628+0000","last_peered":"2024-05-09T05:00:18.652628+0000","last_clean":"2024-05-09T05:00:18.652628+0000","last_became_active":"2024-05-09T05:00:16.880274+0000","last_became_peered":"2024-05-09T05:00:16.880274+0000","last_unstale":"2024-05-09T05:00:18.652628+0000","last_undegraded":"2024-05-09T05:00:18.652628+0000","last_fullsized":"2024-05-09T05:00:18.652628+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:27:18.295238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623650+0000","last_change":"2024-05-09T05:00:12.854427+0000","last_active":"2024-05-09T05:00:19.623650+0000","last_peered":"2024-05-09T05:00:19.623650+0000","last_clean":"2024-05-09T05:00:19.623650+0000","last_became_active":"2024-05-09T05:00:12.854089+0000","last_became_peered":"2024-05-09T05:00:12.854089+0000","last_unstale":"2024-05-09T05:00:19.623650+0000","last_undegraded":"2024-05-09T05:00:19.623650+0000","last_fullsized":"2024-05-09T05:00:19.623650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:33:12.899897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623926+0000","last_change":"2024-05-09T05:00:11.845606+0000","last_active":"2024-05-09T05:00:19.623926+0000","last_peered":"2024-05-09T05:00:19.623926+0000","last_clean":"2024-05-09T05:00:19.623926+0000","last_became_active":"2024-05-09T05:00:11.845227+0000","last_became_peered":"2024-05-09T05:00:11.845227+0000","last_unstale":"2024-05-09T05:00:19.623926+0000","last_undegraded":"2024-05-09T05:00:19.623926+0000","last_fullsized":"2024-05-09T05:00:19.623926+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:50.824636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970860+0000","last_change":"2024-05-09T05:00:16.880664+0000","last_active":"2024-05-09T05:00:19.970860+0000","last_peered":"2024-05-09T05:00:19.970860+0000","last_clean":"2024-05-09T05:00:19.970860+0000","last_became_active":"2024-05-09T05:00:16.880226+0000","last_became_peered":"2024-05-09T05:00:16.880226+0000","last_unstale":"2024-05-09T05:00:19.970860+0000","last_undegraded":"2024-05-09T05:00:19.970860+0000","last_fullsized":"2024-05-09T05:00:19.970860+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:41.016707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271152+0000","last_change":"2024-05-09T05:00:12.858289+0000","last_active":"2024-05-09T05:00:20.271152+0000","last_peered":"2024-05-09T05:00:20.271152+0000","last_clean":"2024-05-09T05:00:20.271152+0000","last_became_active":"2024-05-09T05:00:12.858013+0000","last_became_peered":"2024-05-09T05:00:12.858013+0000","last_unstale":"2024-05-09T05:00:20.271152+0000","last_undegraded":"2024-05-09T05:00:20.271152+0000","last_fullsized":"2024-05-09T05:00:20.271152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:50.968380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:22.574608+0000","last_change":"2024-05-09T05:00:11.842419+0000","last_active":"2024-05-09T05:00:22.574608+0000","last_peered":"2024-05-09T05:00:22.574608+0000","last_clean":"2024-05-09T05:00:22.574608+0000","last_became_active":"2024-05-09T05:00:11.842112+0000","last_became_peered":"2024-05-09T05:00:11.842112+0000","last_unstale":"2024-05-09T05:00:22.574608+0000","last_undegraded":"2024-05-09T05:00:22.574608+0000","last_fullsized":"2024-05-09T05:00:22.574608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:25.803010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271066+0000","last_change":"2024-05-09T05:00:16.874021+0000","last_active":"2024-05-09T05:00:20.271066+0000","last_peered":"2024-05-09T05:00:20.271066+0000","last_clean":"2024-05-09T05:00:20.271066+0000","last_became_active":"2024-05-09T05:00:16.873587+0000","last_became_peered":"2024-05-09T05:00:16.873587+0000","last_unstale":"2024-05-09T05:00:20.271066+0000","last_undegraded":"2024-05-09T05:00:20.271066+0000","last_fullsized":"2024-05-09T05:00:20.271066+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:47:02.206708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882366+0000","last_change":"2024-05-09T05:00:12.844150+0000","last_active":"2024-05-09T05:00:19.882366+0000","last_peered":"2024-05-09T05:00:19.882366+0000","last_clean":"2024-05-09T05:00:19.882366+0000","last_became_active":"2024-05-09T05:00:12.843845+0000","last_became_peered":"2024-05-09T05:00:12.843845+0000","last_unstale":"2024-05-09T05:00:19.882366+0000","last_undegraded":"2024-05-09T05:00:19.882366+0000","last_fullsized":"2024-05-09T05:00:19.882366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:27:58.422255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270961+0000","last_change":"2024-05-09T05:00:11.842387+0000","last_active":"2024-05-09T05:00:20.270961+0000","last_peered":"2024-05-09T05:00:20.270961+0000","last_clean":"2024-05-09T05:00:20.270961+0000","last_became_active":"2024-05-09T05:00:11.840749+0000","last_became_peered":"2024-05-09T05:00:11.840749+0000","last_unstale":"2024-05-09T05:00:20.270961+0000","last_undegraded":"2024-05-09T05:00:20.270961+0000","last_fullsized":"2024-05-09T05:00:20.270961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:27:57.420062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882161+0000","last_change":"2024-05-09T05:00:16.877382+0000","last_active":"2024-05-09T05:00:19.882161+0000","last_peered":"2024-05-09T05:00:19.882161+0000","last_clean":"2024-05-09T05:00:19.882161+0000","last_became_active":"2024-05-09T05:00:16.876966+0000","last_became_peered":"2024-05-09T05:00:16.876966+0000","last_unstale":"2024-05-09T05:00:19.882161+0000","last_undegraded":"2024-05-09T05:00:19.882161+0000","last_fullsized":"2024-05-09T05:00:19.882161+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:20:41.130506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624095+0000","last_change":"2024-05-09T05:00:12.844211+0000","last_active":"2024-05-09T05:00:19.624095+0000","last_peered":"2024-05-09T05:00:19.624095+0000","last_clean":"2024-05-09T05:00:19.624095+0000","last_became_active":"2024-05-09T05:00:12.843545+0000","last_became_peered":"2024-05-09T05:00:12.843545+0000","last_unstale":"2024-05-09T05:00:19.624095+0000","last_undegraded":"2024-05-09T05:00:19.624095+0000","last_fullsized":"2024-05-09T05:00:19.624095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970032+0000","last_change":"2024-05-09T05:00:11.841156+0000","last_active":"2024-05-09T05:00:19.970032+0000","last_peered":"2024-05-09T05:00:19.970032+0000","last_clean":"2024-05-09T05:00:19.970032+0000","last_became_active":"2024-05-09T05:00:11.840620+0000","last_became_peered":"2024-05-09T05:00:11.840620+0000","last_unstale":"2024-05-09T05:00:19.970032+0000","last_undegraded":"2024-05-09T05:00:19.970032+0000","last_fullsized":"2024-05-09T05:00:19.970032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:10.779395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652218+0000","last_change":"2024-05-09T05:00:16.875531+0000","last_active":"2024-05-09T05:00:18.652218+0000","last_peered":"2024-05-09T05:00:18.652218+0000","last_clean":"2024-05-09T05:00:18.652218+0000","last_became_active":"2024-05-09T05:00:16.875056+0000","last_became_peered":"2024-05-09T05:00:16.875056+0000","last_unstale":"2024-05-09T05:00:18.652218+0000","last_undegraded":"2024-05-09T05:00:18.652218+0000","last_fullsized":"2024-05-09T05:00:18.652218+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:41.016707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970758+0000","last_change":"2024-05-09T05:00:12.852295+0000","last_active":"2024-05-09T05:00:19.970758+0000","last_peered":"2024-05-09T05:00:19.970758+0000","last_clean":"2024-05-09T05:00:19.970758+0000","last_became_active":"2024-05-09T05:00:12.851993+0000","last_became_peered":"2024-05-09T05:00:12.851993+0000","last_unstale":"2024-05-09T05:00:19.970758+0000","last_undegraded":"2024-05-09T05:00:19.970758+0000","last_fullsized":"2024-05-09T05:00:19.970758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:44.359533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860982+0000","last_change":"2024-05-09T05:00:11.840063+0000","last_active":"2024-05-09T05:00:17.860982+0000","last_peered":"2024-05-09T05:00:17.860982+0000","last_clean":"2024-05-09T05:00:17.860982+0000","last_became_active":"2024-05-09T05:00:11.839772+0000","last_became_peered":"2024-05-09T05:00:11.839772+0000","last_unstale":"2024-05-09T05:00:17.860982+0000","last_undegraded":"2024-05-09T05:00:17.860982+0000","last_fullsized":"2024-05-09T05:00:17.860982+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:01.738771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652211+0000","last_change":"2024-05-09T05:00:16.883403+0000","last_active":"2024-05-09T05:00:18.652211+0000","last_peered":"2024-05-09T05:00:18.652211+0000","last_clean":"2024-05-09T05:00:18.652211+0000","last_became_active":"2024-05-09T05:00:16.882440+0000","last_became_peered":"2024-05-09T05:00:16.882440+0000","last_unstale":"2024-05-09T05:00:18.652211+0000","last_undegraded":"2024-05-09T05:00:18.652211+0000","last_fullsized":"2024-05-09T05:00:18.652211+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:18:15.413800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624058+0000","last_change":"2024-05-09T05:00:16.883006+0000","last_active":"2024-05-09T05:00:19.624058+0000","last_peered":"2024-05-09T05:00:19.624058+0000","last_clean":"2024-05-09T05:00:19.624058+0000","last_became_active":"2024-05-09T05:00:16.882624+0000","last_became_peered":"2024-05-09T05:00:16.882624+0000","last_unstale":"2024-05-09T05:00:19.624058+0000","last_undegraded":"2024-05-09T05:00:19.624058+0000","last_fullsized":"2024-05-09T05:00:19.624058+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:02.171370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270485+0000","last_change":"2024-05-09T05:00:12.850411+0000","last_active":"2024-05-09T05:00:20.270485+0000","last_peered":"2024-05-09T05:00:20.270485+0000","last_clean":"2024-05-09T05:00:20.270485+0000","last_became_active":"2024-05-09T05:00:12.850121+0000","last_became_peered":"2024-05-09T05:00:12.850121+0000","last_unstale":"2024-05-09T05:00:20.270485+0000","last_undegraded":"2024-05-09T05:00:20.270485+0000","last_fullsized":"2024-05-09T05:00:20.270485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:30.691332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970418+0000","last_change":"2024-05-09T05:00:11.843746+0000","last_active":"2024-05-09T05:00:19.970418+0000","last_peered":"2024-05-09T05:00:19.970418+0000","last_clean":"2024-05-09T05:00:19.970418+0000","last_became_active":"2024-05-09T05:00:11.843423+0000","last_became_peered":"2024-05-09T05:00:11.843423+0000","last_unstale":"2024-05-09T05:00:19.970418+0000","last_undegraded":"2024-05-09T05:00:19.970418+0000","last_fullsized":"2024-05-09T05:00:19.970418+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:41:27.364045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860975+0000","last_change":"2024-05-09T05:00:16.873412+0000","last_active":"2024-05-09T05:00:17.860975+0000","last_peered":"2024-05-09T05:00:17.860975+0000","last_clean":"2024-05-09T05:00:17.860975+0000","last_became_active":"2024-05-09T05:00:16.873014+0000","last_became_peered":"2024-05-09T05:00:16.873014+0000","last_unstale":"2024-05-09T05:00:17.860975+0000","last_undegraded":"2024-05-09T05:00:17.860975+0000","last_fullsized":"2024-05-09T05:00:17.860975+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:20.415639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652208+0000","last_change":"2024-05-09T05:00:12.856243+0000","last_active":"2024-05-09T05:00:18.652208+0000","last_peered":"2024-05-09T05:00:18.652208+0000","last_clean":"2024-05-09T05:00:18.652208+0000","last_became_active":"2024-05-09T05:00:12.854306+0000","last_became_peered":"2024-05-09T05:00:12.854306+0000","last_unstale":"2024-05-09T05:00:18.652208+0000","last_undegraded":"2024-05-09T05:00:18.652208+0000","last_fullsized":"2024-05-09T05:00:18.652208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:10:10.149613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652448+0000","last_change":"2024-05-09T05:00:11.837046+0000","last_active":"2024-05-09T05:00:18.652448+0000","last_peered":"2024-05-09T05:00:18.652448+0000","last_clean":"2024-05-09T05:00:18.652448+0000","last_became_active":"2024-05-09T05:00:11.836773+0000","last_became_peered":"2024-05-09T05:00:11.836773+0000","last_unstale":"2024-05-09T05:00:18.652448+0000","last_undegraded":"2024-05-09T05:00:18.652448+0000","last_fullsized":"2024-05-09T05:00:18.652448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:48:26.640184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860385+0000","last_change":"2024-05-09T05:00:11.841442+0000","last_active":"2024-05-09T05:00:17.860385+0000","last_peered":"2024-05-09T05:00:17.860385+0000","last_clean":"2024-05-09T05:00:17.860385+0000","last_became_active":"2024-05-09T05:00:11.841099+0000","last_became_peered":"2024-05-09T05:00:11.841099+0000","last_unstale":"2024-05-09T05:00:17.860385+0000","last_undegraded":"2024-05-09T05:00:17.860385+0000","last_fullsized":"2024-05-09T05:00:17.860385+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:41:27.364045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623878+0000","last_change":"2024-05-09T05:00:12.845049+0000","last_active":"2024-05-09T05:00:19.623878+0000","last_peered":"2024-05-09T05:00:19.623878+0000","last_clean":"2024-05-09T05:00:19.623878+0000","last_became_active":"2024-05-09T05:00:12.844724+0000","last_became_peered":"2024-05-09T05:00:12.844724+0000","last_unstale":"2024-05-09T05:00:19.623878+0000","last_undegraded":"2024-05-09T05:00:19.623878+0000","last_fullsized":"2024-05-09T05:00:19.623878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:45.463031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860933+0000","last_change":"2024-05-09T05:00:16.883497+0000","last_active":"2024-05-09T05:00:17.860933+0000","last_peered":"2024-05-09T05:00:17.860933+0000","last_clean":"2024-05-09T05:00:17.860933+0000","last_became_active":"2024-05-09T05:00:16.883063+0000","last_became_peered":"2024-05-09T05:00:16.883063+0000","last_unstale":"2024-05-09T05:00:17.860933+0000","last_undegraded":"2024-05-09T05:00:17.860933+0000","last_fullsized":"2024-05-09T05:00:17.860933+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:04:58.832049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652723+0000","last_change":"2024-05-09T05:00:11.847505+0000","last_active":"2024-05-09T05:00:18.652723+0000","last_peered":"2024-05-09T05:00:18.652723+0000","last_clean":"2024-05-09T05:00:18.652723+0000","last_became_active":"2024-05-09T05:00:11.847187+0000","last_became_peered":"2024-05-09T05:00:11.847187+0000","last_unstale":"2024-05-09T05:00:18.652723+0000","last_undegraded":"2024-05-09T05:00:18.652723+0000","last_fullsized":"2024-05-09T05:00:18.652723+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:50.824636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882122+0000","last_change":"2024-05-09T05:00:12.838998+0000","last_active":"2024-05-09T05:00:19.882122+0000","last_peered":"2024-05-09T05:00:19.882122+0000","last_clean":"2024-05-09T05:00:19.882122+0000","last_became_active":"2024-05-09T05:00:12.838702+0000","last_became_peered":"2024-05-09T05:00:12.838702+0000","last_unstale":"2024-05-09T05:00:19.882122+0000","last_undegraded":"2024-05-09T05:00:19.882122+0000","last_fullsized":"2024-05-09T05:00:19.882122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:50.662283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652939+0000","last_change":"2024-05-09T05:00:16.876805+0000","last_active":"2024-05-09T05:00:18.652939+0000","last_peered":"2024-05-09T05:00:18.652939+0000","last_clean":"2024-05-09T05:00:18.652939+0000","last_became_active":"2024-05-09T05:00:16.876265+0000","last_became_peered":"2024-05-09T05:00:16.876265+0000","last_unstale":"2024-05-09T05:00:18.652939+0000","last_undegraded":"2024-05-09T05:00:18.652939+0000","last_fullsized":"2024-05-09T05:00:18.652939+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:54:27.622037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860099+0000","last_change":"2024-05-09T05:00:12.857527+0000","last_active":"2024-05-09T05:00:17.860099+0000","last_peered":"2024-05-09T05:00:17.860099+0000","last_clean":"2024-05-09T05:00:17.860099+0000","last_became_active":"2024-05-09T05:00:12.857202+0000","last_became_peered":"2024-05-09T05:00:12.857202+0000","last_unstale":"2024-05-09T05:00:17.860099+0000","last_undegraded":"2024-05-09T05:00:17.860099+0000","last_fullsized":"2024-05-09T05:00:17.860099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:45.463031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860631+0000","last_change":"2024-05-09T05:00:12.844334+0000","last_active":"2024-05-09T05:00:17.860631+0000","last_peered":"2024-05-09T05:00:17.860631+0000","last_clean":"2024-05-09T05:00:17.860631+0000","last_became_active":"2024-05-09T05:00:12.844018+0000","last_became_peered":"2024-05-09T05:00:12.844018+0000","last_unstale":"2024-05-09T05:00:17.860631+0000","last_undegraded":"2024-05-09T05:00:17.860631+0000","last_fullsized":"2024-05-09T05:00:17.860631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:40.797004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970591+0000","last_change":"2024-05-09T05:00:12.855985+0000","last_active":"2024-05-09T05:00:19.970591+0000","last_peered":"2024-05-09T05:00:19.970591+0000","last_clean":"2024-05-09T05:00:19.970591+0000","last_became_active":"2024-05-09T05:00:12.855586+0000","last_became_peered":"2024-05-09T05:00:12.855586+0000","last_unstale":"2024-05-09T05:00:19.970591+0000","last_undegraded":"2024-05-09T05:00:19.970591+0000","last_fullsized":"2024-05-09T05:00:19.970591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:06.793812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623781+0000","last_change":"2024-05-09T05:00:12.848323+0000","last_active":"2024-05-09T05:00:19.623781+0000","last_peered":"2024-05-09T05:00:19.623781+0000","last_clean":"2024-05-09T05:00:19.623781+0000","last_became_active":"2024-05-09T05:00:12.848053+0000","last_became_peered":"2024-05-09T05:00:12.848053+0000","last_unstale":"2024-05-09T05:00:19.623781+0000","last_undegraded":"2024-05-09T05:00:19.623781+0000","last_fullsized":"2024-05-09T05:00:19.623781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:19.080163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652749+0000","last_change":"2024-05-09T05:00:12.849804+0000","last_active":"2024-05-09T05:00:18.652749+0000","last_peered":"2024-05-09T05:00:18.652749+0000","last_clean":"2024-05-09T05:00:18.652749+0000","last_became_active":"2024-05-09T05:00:12.849530+0000","last_became_peered":"2024-05-09T05:00:12.849530+0000","last_unstale":"2024-05-09T05:00:18.652749+0000","last_undegraded":"2024-05-09T05:00:18.652749+0000","last_fullsized":"2024-05-09T05:00:18.652749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:20:37.102430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270858+0000","last_change":"2024-05-09T05:00:12.848438+0000","last_active":"2024-05-09T05:00:20.270858+0000","last_peered":"2024-05-09T05:00:20.270858+0000","last_clean":"2024-05-09T05:00:20.270858+0000","last_became_active":"2024-05-09T05:00:12.848176+0000","last_became_peered":"2024-05-09T05:00:12.848176+0000","last_unstale":"2024-05-09T05:00:20.270858+0000","last_undegraded":"2024-05-09T05:00:20.270858+0000","last_fullsized":"2024-05-09T05:00:20.270858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860997+0000","last_change":"2024-05-09T05:00:12.855310+0000","last_active":"2024-05-09T05:00:17.860997+0000","last_peered":"2024-05-09T05:00:17.860997+0000","last_clean":"2024-05-09T05:00:17.860997+0000","last_became_active":"2024-05-09T05:00:12.854031+0000","last_became_peered":"2024-05-09T05:00:12.854031+0000","last_unstale":"2024-05-09T05:00:17.860997+0000","last_undegraded":"2024-05-09T05:00:17.860997+0000","last_fullsized":"2024-05-09T05:00:17.860997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:44.359533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652800+0000","last_change":"2024-05-09T05:00:12.849125+0000","last_active":"2024-05-09T05:00:18.652800+0000","last_peered":"2024-05-09T05:00:18.652800+0000","last_clean":"2024-05-09T05:00:18.652800+0000","last_became_active":"2024-05-09T05:00:12.848852+0000","last_became_peered":"2024-05-09T05:00:12.848852+0000","last_unstale":"2024-05-09T05:00:18.652800+0000","last_undegraded":"2024-05-09T05:00:18.652800+0000","last_fullsized":"2024-05-09T05:00:18.652800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970301+0000","last_change":"2024-05-09T05:00:12.847892+0000","last_active":"2024-05-09T05:00:19.970301+0000","last_peered":"2024-05-09T05:00:19.970301+0000","last_clean":"2024-05-09T05:00:19.970301+0000","last_became_active":"2024-05-09T05:00:12.847051+0000","last_became_peered":"2024-05-09T05:00:12.847051+0000","last_unstale":"2024-05-09T05:00:19.970301+0000","last_undegraded":"2024-05-09T05:00:19.970301+0000","last_fullsized":"2024-05-09T05:00:19.970301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652323+0000","last_change":"2024-05-09T05:00:12.848463+0000","last_active":"2024-05-09T05:00:18.652323+0000","last_peered":"2024-05-09T05:00:18.652323+0000","last_clean":"2024-05-09T05:00:18.652323+0000","last_became_active":"2024-05-09T05:00:12.848170+0000","last_became_peered":"2024-05-09T05:00:12.848170+0000","last_unstale":"2024-05-09T05:00:18.652323+0000","last_undegraded":"2024-05-09T05:00:18.652323+0000","last_fullsized":"2024-05-09T05:00:18.652323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:45.463031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970492+0000","last_change":"2024-05-09T05:00:12.852750+0000","last_active":"2024-05-09T05:00:19.970492+0000","last_peered":"2024-05-09T05:00:19.970492+0000","last_clean":"2024-05-09T05:00:19.970492+0000","last_became_active":"2024-05-09T05:00:12.852487+0000","last_became_peered":"2024-05-09T05:00:12.852487+0000","last_unstale":"2024-05-09T05:00:19.970492+0000","last_undegraded":"2024-05-09T05:00:19.970492+0000","last_fullsized":"2024-05-09T05:00:19.970492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:50.968380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623652+0000","last_change":"2024-05-09T05:00:12.850281+0000","last_active":"2024-05-09T05:00:19.623652+0000","last_peered":"2024-05-09T05:00:19.623652+0000","last_clean":"2024-05-09T05:00:19.623652+0000","last_became_active":"2024-05-09T05:00:12.849875+0000","last_became_peered":"2024-05-09T05:00:12.849875+0000","last_unstale":"2024-05-09T05:00:19.623652+0000","last_undegraded":"2024-05-09T05:00:19.623652+0000","last_fullsized":"2024-05-09T05:00:19.623652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:10:10.149613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860536+0000","last_change":"2024-05-09T05:00:12.841388+0000","last_active":"2024-05-09T05:00:17.860536+0000","last_peered":"2024-05-09T05:00:17.860536+0000","last_clean":"2024-05-09T05:00:17.860536+0000","last_became_active":"2024-05-09T05:00:12.841121+0000","last_became_peered":"2024-05-09T05:00:12.841121+0000","last_unstale":"2024-05-09T05:00:17.860536+0000","last_undegraded":"2024-05-09T05:00:17.860536+0000","last_fullsized":"2024-05-09T05:00:17.860536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:30.691332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271354+0000","last_change":"2024-05-09T05:00:12.853880+0000","last_active":"2024-05-09T05:00:20.271354+0000","last_peered":"2024-05-09T05:00:20.271354+0000","last_clean":"2024-05-09T05:00:20.271354+0000","last_became_active":"2024-05-09T05:00:12.852044+0000","last_became_peered":"2024-05-09T05:00:12.852044+0000","last_unstale":"2024-05-09T05:00:20.271354+0000","last_undegraded":"2024-05-09T05:00:20.271354+0000","last_fullsized":"2024-05-09T05:00:20.271354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:31:56.397467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860942+0000","last_change":"2024-05-09T05:00:12.842477+0000","last_active":"2024-05-09T05:00:17.860942+0000","last_peered":"2024-05-09T05:00:17.860942+0000","last_clean":"2024-05-09T05:00:17.860942+0000","last_became_active":"2024-05-09T05:00:12.842135+0000","last_became_peered":"2024-05-09T05:00:12.842135+0000","last_unstale":"2024-05-09T05:00:17.860942+0000","last_undegraded":"2024-05-09T05:00:17.860942+0000","last_fullsized":"2024-05-09T05:00:17.860942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:27:58.422255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.971029+0000","last_change":"2024-05-09T05:00:12.852559+0000","last_active":"2024-05-09T05:00:19.971029+0000","last_peered":"2024-05-09T05:00:19.971029+0000","last_clean":"2024-05-09T05:00:19.971029+0000","last_became_active":"2024-05-09T05:00:12.852370+0000","last_became_peered":"2024-05-09T05:00:12.852370+0000","last_unstale":"2024-05-09T05:00:19.971029+0000","last_undegraded":"2024-05-09T05:00:19.971029+0000","last_fullsized":"2024-05-09T05:00:19.971029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:58.178632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970409+0000","last_change":"2024-05-09T05:00:12.856404+0000","last_active":"2024-05-09T05:00:19.970409+0000","last_peered":"2024-05-09T05:00:19.970409+0000","last_clean":"2024-05-09T05:00:19.970409+0000","last_became_active":"2024-05-09T05:00:12.856126+0000","last_became_peered":"2024-05-09T05:00:12.856126+0000","last_unstale":"2024-05-09T05:00:19.970409+0000","last_undegraded":"2024-05-09T05:00:19.970409+0000","last_fullsized":"2024-05-09T05:00:19.970409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270574+0000","last_change":"2024-05-09T05:00:12.847697+0000","last_active":"2024-05-09T05:00:20.270574+0000","last_peered":"2024-05-09T05:00:20.270574+0000","last_clean":"2024-05-09T05:00:20.270574+0000","last_became_active":"2024-05-09T05:00:12.847412+0000","last_became_peered":"2024-05-09T05:00:12.847412+0000","last_unstale":"2024-05-09T05:00:20.270574+0000","last_undegraded":"2024-05-09T05:00:20.270574+0000","last_fullsized":"2024-05-09T05:00:20.270574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:19.080163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623651+0000","last_change":"2024-05-09T05:00:12.846578+0000","last_active":"2024-05-09T05:00:19.623651+0000","last_peered":"2024-05-09T05:00:19.623651+0000","last_clean":"2024-05-09T05:00:19.623651+0000","last_became_active":"2024-05-09T05:00:12.844465+0000","last_became_peered":"2024-05-09T05:00:12.844465+0000","last_unstale":"2024-05-09T05:00:19.623651+0000","last_undegraded":"2024-05-09T05:00:19.623651+0000","last_fullsized":"2024-05-09T05:00:19.623651+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:17:11.682946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652503+0000","last_change":"2024-05-09T05:00:12.852179+0000","last_active":"2024-05-09T05:00:18.652503+0000","last_peered":"2024-05-09T05:00:18.652503+0000","last_clean":"2024-05-09T05:00:18.652503+0000","last_became_active":"2024-05-09T05:00:12.849715+0000","last_became_peered":"2024-05-09T05:00:12.849715+0000","last_unstale":"2024-05-09T05:00:18.652503+0000","last_undegraded":"2024-05-09T05:00:18.652503+0000","last_fullsized":"2024-05-09T05:00:18.652503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:19.080163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882233+0000","last_change":"2024-05-09T05:00:12.837696+0000","last_active":"2024-05-09T05:00:19.882233+0000","last_peered":"2024-05-09T05:00:19.882233+0000","last_clean":"2024-05-09T05:00:19.882233+0000","last_became_active":"2024-05-09T05:00:12.837445+0000","last_became_peered":"2024-05-09T05:00:12.837445+0000","last_unstale":"2024-05-09T05:00:19.882233+0000","last_undegraded":"2024-05-09T05:00:19.882233+0000","last_fullsized":"2024-05-09T05:00:19.882233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:40.797004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860739+0000","last_change":"2024-05-09T05:00:12.840693+0000","last_active":"2024-05-09T05:00:17.860739+0000","last_peered":"2024-05-09T05:00:17.860739+0000","last_clean":"2024-05-09T05:00:17.860739+0000","last_became_active":"2024-05-09T05:00:12.840400+0000","last_became_peered":"2024-05-09T05:00:12.840400+0000","last_unstale":"2024-05-09T05:00:17.860739+0000","last_undegraded":"2024-05-09T05:00:17.860739+0000","last_fullsized":"2024-05-09T05:00:17.860739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860995+0000","last_change":"2024-05-09T05:00:12.841318+0000","last_active":"2024-05-09T05:00:17.860995+0000","last_peered":"2024-05-09T05:00:17.860995+0000","last_clean":"2024-05-09T05:00:17.860995+0000","last_became_active":"2024-05-09T05:00:12.841026+0000","last_became_peered":"2024-05-09T05:00:12.841026+0000","last_unstale":"2024-05-09T05:00:17.860995+0000","last_undegraded":"2024-05-09T05:00:17.860995+0000","last_fullsized":"2024-05-09T05:00:17.860995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271552+0000","last_change":"2024-05-09T05:00:12.852840+0000","last_active":"2024-05-09T05:00:20.271552+0000","last_peered":"2024-05-09T05:00:20.271552+0000","last_clean":"2024-05-09T05:00:20.271552+0000","last_became_active":"2024-05-09T05:00:12.852259+0000","last_became_peered":"2024-05-09T05:00:12.852259+0000","last_unstale":"2024-05-09T05:00:20.271552+0000","last_undegraded":"2024-05-09T05:00:20.271552+0000","last_fullsized":"2024-05-09T05:00:20.271552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:16:58.143294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652120+0000","last_change":"2024-05-09T05:00:12.846320+0000","last_active":"2024-05-09T05:00:18.652120+0000","last_peered":"2024-05-09T05:00:18.652120+0000","last_clean":"2024-05-09T05:00:18.652120+0000","last_became_active":"2024-05-09T05:00:12.845732+0000","last_became_peered":"2024-05-09T05:00:12.845732+0000","last_unstale":"2024-05-09T05:00:18.652120+0000","last_undegraded":"2024-05-09T05:00:18.652120+0000","last_fullsized":"2024-05-09T05:00:18.652120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:33:12.899897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270936+0000","last_change":"2024-05-09T05:00:12.858255+0000","last_active":"2024-05-09T05:00:20.270936+0000","last_peered":"2024-05-09T05:00:20.270936+0000","last_clean":"2024-05-09T05:00:20.270936+0000","last_became_active":"2024-05-09T05:00:12.857971+0000","last_became_peered":"2024-05-09T05:00:12.857971+0000","last_unstale":"2024-05-09T05:00:20.270936+0000","last_undegraded":"2024-05-09T05:00:20.270936+0000","last_fullsized":"2024-05-09T05:00:20.270936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:23.066586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970422+0000","last_change":"2024-05-09T05:00:12.846491+0000","last_active":"2024-05-09T05:00:19.970422+0000","last_peered":"2024-05-09T05:00:19.970422+0000","last_clean":"2024-05-09T05:00:19.970422+0000","last_became_active":"2024-05-09T05:00:12.845643+0000","last_became_peered":"2024-05-09T05:00:12.845643+0000","last_unstale":"2024-05-09T05:00:19.970422+0000","last_undegraded":"2024-05-09T05:00:19.970422+0000","last_fullsized":"2024-05-09T05:00:19.970422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:20:37.102430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270963+0000","last_change":"2024-05-09T05:00:12.854820+0000","last_active":"2024-05-09T05:00:20.270963+0000","last_peered":"2024-05-09T05:00:20.270963+0000","last_clean":"2024-05-09T05:00:20.270963+0000","last_became_active":"2024-05-09T05:00:12.853663+0000","last_became_peered":"2024-05-09T05:00:12.853663+0000","last_unstale":"2024-05-09T05:00:20.270963+0000","last_undegraded":"2024-05-09T05:00:20.270963+0000","last_fullsized":"2024-05-09T05:00:20.270963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:04.977127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970662+0000","last_change":"2024-05-09T05:00:12.855723+0000","last_active":"2024-05-09T05:00:19.970662+0000","last_peered":"2024-05-09T05:00:19.970662+0000","last_clean":"2024-05-09T05:00:19.970662+0000","last_became_active":"2024-05-09T05:00:12.855479+0000","last_became_peered":"2024-05-09T05:00:12.855479+0000","last_unstale":"2024-05-09T05:00:19.970662+0000","last_undegraded":"2024-05-09T05:00:19.970662+0000","last_fullsized":"2024-05-09T05:00:19.970662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:01:08.488591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623805+0000","last_change":"2024-05-09T05:00:12.838168+0000","last_active":"2024-05-09T05:00:19.623805+0000","last_peered":"2024-05-09T05:00:19.623805+0000","last_clean":"2024-05-09T05:00:19.623805+0000","last_became_active":"2024-05-09T05:00:12.837892+0000","last_became_peered":"2024-05-09T05:00:12.837892+0000","last_unstale":"2024-05-09T05:00:19.623805+0000","last_undegraded":"2024-05-09T05:00:19.623805+0000","last_fullsized":"2024-05-09T05:00:19.623805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:30.691332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271359+0000","last_change":"2024-05-09T05:00:12.851393+0000","last_active":"2024-05-09T05:00:20.271359+0000","last_peered":"2024-05-09T05:00:20.271359+0000","last_clean":"2024-05-09T05:00:20.271359+0000","last_became_active":"2024-05-09T05:00:12.851057+0000","last_became_peered":"2024-05-09T05:00:12.851057+0000","last_unstale":"2024-05-09T05:00:20.271359+0000","last_undegraded":"2024-05-09T05:00:20.271359+0000","last_fullsized":"2024-05-09T05:00:20.271359+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271442+0000","last_change":"2024-05-09T05:00:12.853084+0000","last_active":"2024-05-09T05:00:20.271442+0000","last_peered":"2024-05-09T05:00:20.271442+0000","last_clean":"2024-05-09T05:00:20.271442+0000","last_became_active":"2024-05-09T05:00:12.852602+0000","last_became_peered":"2024-05-09T05:00:12.852602+0000","last_unstale":"2024-05-09T05:00:20.271442+0000","last_undegraded":"2024-05-09T05:00:20.271442+0000","last_fullsized":"2024-05-09T05:00:20.271442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:58.178632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:00:33.021 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:00:33.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.160+0000 7f3cf8409640 1 Processor -- start 2024-05-09T05:00:33.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.160+0000 7f3cf8409640 1 -- start start 2024-05-09T05:00:33.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.161+0000 7f3cf8409640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cf00b4d30 0x7f3cf0169ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:33.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.161+0000 7f3cf8409640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cf015f7a0 con 0x7f3cf015f3c0 2024-05-09T05:00:33.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.161+0000 7f3cf8409640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cf009f480 con 0x7f3cf0073fb0 2024-05-09T05:00:33.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.161+0000 7f3cf8409640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f3cf0071cb0 con 0x7f3cf00b4d30 2024-05-09T05:00:33.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.161+0000 7f3cf61a5640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3cf015f3c0 0x7f3cf009ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38402/0 (socket says 172.21.15.60:38402) 2024-05-09T05:00:33.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.161+0000 7f3cf61a5640 1 -- 172.21.15.60:0/3570096968 learned_addr learned my addr 172.21.15.60:0/3570096968 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:33.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.161+0000 7f3cf69a6640 1 --2- 172.21.15.60:0/3570096968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cf00b4d30 0x7f3cf0169ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:33.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.161+0000 7f3cf51a3640 1 -- 172.21.15.60:0/3570096968 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2856746094 0 0) 0x7f3cf015f7a0 con 0x7f3cf015f3c0 2024-05-09T05:00:33.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.162+0000 7f3cf51a3640 1 -- 172.21.15.60:0/3570096968 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cd8003520 con 0x7f3cf015f3c0 2024-05-09T05:00:33.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.162+0000 7f3cf51a3640 1 -- 172.21.15.60:0/3570096968 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1222825071 0 0) 0x7f3cf009f480 con 0x7f3cf0073fb0 2024-05-09T05:00:33.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.162+0000 7f3cf51a3640 1 -- 172.21.15.60:0/3570096968 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cf015f7a0 con 0x7f3cf0073fb0 2024-05-09T05:00:33.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.162+0000 7f3cf51a3640 1 -- 172.21.15.60:0/3570096968 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3664593703 0 0) 0x7f3cd8003520 con 0x7f3cf015f3c0 2024-05-09T05:00:33.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.162+0000 7f3cf69a6640 1 -- 172.21.15.60:0/3570096968 >> v1:172.21.15.114:6789/0 conn(0x7f3cf0073fb0 legacy=0x7f3cf00b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.162+0000 7f3cf69a6640 1 -- 172.21.15.60:0/3570096968 >> v1:172.21.15.60:6789/0 conn(0x7f3cf015f3c0 legacy=0x7f3cf009ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.162+0000 7f3cf69a6640 1 -- 172.21.15.60:0/3570096968 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf009f760 con 0x7f3cf00b4d30 2024-05-09T05:00:33.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.163+0000 7f3cf69a6640 1 --2- 172.21.15.60:0/3570096968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cf00b4d30 0x7f3cf0169ef0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3ce80097d0 tx=0x7f3ce802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfd324c07ed9f319 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:33.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.163+0000 7f3cf51a3640 1 -- 172.21.15.60:0/3570096968 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8037070 con 0x7f3cf00b4d30 2024-05-09T05:00:33.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.163+0000 7f3cf51a3640 1 -- 172.21.15.60:0/3570096968 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ce802ab90 con 0x7f3cf00b4d30 2024-05-09T05:00:33.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.163+0000 7f3cf51a3640 1 -- 172.21.15.60:0/3570096968 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce0001a70 con 0x7f3cf00b4d30 2024-05-09T05:00:33.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.163+0000 7f3cf8409640 1 -- 172.21.15.60:0/3570096968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cf00b4d30 msgr2=0x7f3cf0169ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.163+0000 7f3cf8409640 1 --2- 172.21.15.60:0/3570096968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cf00b4d30 0x7f3cf0169ef0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3ce80097d0 tx=0x7f3ce802a100 comp rx=0 tx=0).stop 2024-05-09T05:00:33.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.163+0000 7f3cf8409640 1 -- 172.21.15.60:0/3570096968 shutdown_connections 2024-05-09T05:00:33.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.163+0000 7f3cf8409640 1 --2- 172.21.15.60:0/3570096968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cf00b4d30 0x7f3cf0169ef0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.163+0000 7f3cf8409640 1 -- 172.21.15.60:0/3570096968 >> 172.21.15.60:0/3570096968 conn(0x7f3cf00d1470 msgr2=0x7f3cf00d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:33.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.163+0000 7f3cf8409640 1 -- 172.21.15.60:0/3570096968 shutdown_connections 2024-05-09T05:00:33.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.163+0000 7f3cf8409640 1 -- 172.21.15.60:0/3570096968 wait complete. 2024-05-09T05:00:33.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.165+0000 7f3cf8409640 1 Processor -- start 2024-05-09T05:00:33.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.165+0000 7f3cf8409640 1 -- start start 2024-05-09T05:00:33.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.165+0000 7f3cf8409640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cf0073fb0 0x7f3cf00b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:33.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.165+0000 7f3cf8409640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cf00b4d30 0x7f3cf01572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:33.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.165+0000 7f3cf8409640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cf015f3c0 0x7f3cf0158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:33.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.165+0000 7f3cf8409640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf00732d0 con 0x7f3cf00b4d30 2024-05-09T05:00:33.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.165+0000 7f3cf8409640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf0073150 con 0x7f3cf015f3c0 2024-05-09T05:00:33.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.165+0000 7f3cf8409640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f3cf0073450 con 0x7f3cf0073fb0 2024-05-09T05:00:33.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.165+0000 7f3cf59a4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cf00b4d30 0x7f3cf01572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:33.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.166+0000 7f3cf59a4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cf00b4d30 0x7f3cf01572e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34772/0 (socket says 172.21.15.60:34772) 2024-05-09T05:00:33.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.166+0000 7f3cf59a4640 1 -- 172.21.15.60:0/1564030929 learned_addr learned my addr 172.21.15.60:0/1564030929 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:33.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.166+0000 7f3cf69a6640 1 --2- 172.21.15.60:0/1564030929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cf015f3c0 0x7f3cf0158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:33.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.166+0000 7f3cf61a5640 1 --2- 172.21.15.60:0/1564030929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cf0073fb0 0x7f3cf00b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:33.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.166+0000 7f3cf59a4640 1 -- 172.21.15.60:0/1564030929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cf0073fb0 msgr2=0x7f3cf00b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.166+0000 7f3cf59a4640 1 --2- 172.21.15.60:0/1564030929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cf0073fb0 0x7f3cf00b2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.166+0000 7f3cf59a4640 1 -- 172.21.15.60:0/1564030929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cf015f3c0 msgr2=0x7f3cf0158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.166+0000 7f3cf59a4640 1 --2- 172.21.15.60:0/1564030929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cf015f3c0 0x7f3cf0158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.166+0000 7f3cf59a4640 1 -- 172.21.15.60:0/1564030929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf0157cc0 con 0x7f3cf00b4d30 2024-05-09T05:00:33.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.166+0000 7f3cf59a4640 1 --2- 172.21.15.60:0/1564030929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cf00b4d30 0x7f3cf01572e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3ce400c490 tx=0x7f3ce400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:33.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.166+0000 7f3ceeffd640 1 -- 172.21.15.60:0/1564030929 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce40052d0 con 0x7f3cf00b4d30 2024-05-09T05:00:33.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.166+0000 7f3ceeffd640 1 -- 172.21.15.60:0/1564030929 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ce4005490 con 0x7f3cf00b4d30 2024-05-09T05:00:33.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.166+0000 7f3cf8409640 1 -- 172.21.15.60:0/1564030929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf0157fa0 con 0x7f3cf00b4d30 2024-05-09T05:00:33.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.166+0000 7f3ceeffd640 1 -- 172.21.15.60:0/1564030929 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4012870 con 0x7f3cf00b4d30 2024-05-09T05:00:33.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.166+0000 7f3cf8409640 1 -- 172.21.15.60:0/1564030929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cf0158560 con 0x7f3cf00b4d30 2024-05-09T05:00:33.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.167+0000 7f3cf8409640 1 -- 172.21.15.60:0/1564030929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cf0051040 con 0x7f3cf00b4d30 2024-05-09T05:00:33.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.168+0000 7f3ceeffd640 1 -- 172.21.15.60:0/1564030929 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3ce4012a10 con 0x7f3cf00b4d30 2024-05-09T05:00:33.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.168+0000 7f3ceeffd640 1 --2- 172.21.15.60:0/1564030929 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f3cdc0762d0 0x7f3cdc078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:33.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.168+0000 7f3ceeffd640 1 -- 172.21.15.60:0/1564030929 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3ce4097580 con 0x7f3cf00b4d30 2024-05-09T05:00:33.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.169+0000 7f3cf61a5640 1 --2- 172.21.15.60:0/1564030929 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f3cdc0762d0 0x7f3cdc078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:33.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.169+0000 7f3cf61a5640 1 --2- 172.21.15.60:0/1564030929 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f3cdc0762d0 0x7f3cdc078790 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0000f80 tx=0x7f3ce0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:33.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.173+0000 7f3ceeffd640 1 -- 172.21.15.60:0/1564030929 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3ce4060430 con 0x7f3cf00b4d30 2024-05-09T05:00:33.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.359+0000 7f3cf8409640 1 -- 172.21.15.60:0/1564030929 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3cf00b2fe0 con 0x7f3cdc0762d0 2024-05-09T05:00:33.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.368+0000 7f3ceeffd640 1 -- 172.21.15.60:0/1564030929 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332523 (secure 0 0 0) 0x7f3cf00b2fe0 con 0x7f3cdc0762d0 2024-05-09T05:00:33.372 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:33.374 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-05-09T05:00:33.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.371+0000 7f3cf8409640 1 -- 172.21.15.60:0/1564030929 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f3cdc0762d0 msgr2=0x7f3cdc078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.371+0000 7f3cf8409640 1 --2- 172.21.15.60:0/1564030929 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f3cdc0762d0 0x7f3cdc078790 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0000f80 tx=0x7f3ce0009040 comp rx=0 tx=0).stop 2024-05-09T05:00:33.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.371+0000 7f3cf8409640 1 -- 172.21.15.60:0/1564030929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cf00b4d30 msgr2=0x7f3cf01572e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.371+0000 7f3cf8409640 1 --2- 172.21.15.60:0/1564030929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cf00b4d30 0x7f3cf01572e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3ce400c490 tx=0x7f3ce400c950 comp rx=0 tx=0).stop 2024-05-09T05:00:33.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.371+0000 7f3cf8409640 1 -- 172.21.15.60:0/1564030929 shutdown_connections 2024-05-09T05:00:33.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.371+0000 7f3cf8409640 1 --2- 172.21.15.60:0/1564030929 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f3cdc0762d0 0x7f3cdc078790 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.371+0000 7f3cf8409640 1 --2- 172.21.15.60:0/1564030929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3cf015f3c0 0x7f3cf0158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.371+0000 7f3cf8409640 1 --2- 172.21.15.60:0/1564030929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cf00b4d30 0x7f3cf01572e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.371+0000 7f3cf8409640 1 --2- 172.21.15.60:0/1564030929 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3cf0073fb0 0x7f3cf00b2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.371+0000 7f3cf8409640 1 -- 172.21.15.60:0/1564030929 >> 172.21.15.60:0/1564030929 conn(0x7f3cf00d1470 msgr2=0x7f3cf009e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:33.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.371+0000 7f3cf8409640 1 -- 172.21.15.60:0/1564030929 shutdown_connections 2024-05-09T05:00:33.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.371+0000 7f3cf8409640 1 -- 172.21.15.60:0/1564030929 wait complete. 2024-05-09T05:00:33.383 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-05-09T05:00:32.915129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223524,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751196,"statfs":{"total":773094113280,"available":772865224704,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003187"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624117+0000","last_change":"2024-05-09T05:00:11.849304+0000","last_active":"2024-05-09T05:00:19.624117+0000","last_peered":"2024-05-09T05:00:19.624117+0000","last_clean":"2024-05-09T05:00:19.624117+0000","last_became_active":"2024-05-09T05:00:11.848977+0000","last_became_peered":"2024-05-09T05:00:11.848977+0000","last_unstale":"2024-05-09T05:00:19.624117+0000","last_undegraded":"2024-05-09T05:00:19.624117+0000","last_fullsized":"2024-05-09T05:00:19.624117+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:48:26.640184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652587+0000","last_change":"2024-05-09T05:00:12.853461+0000","last_active":"2024-05-09T05:00:18.652587+0000","last_peered":"2024-05-09T05:00:18.652587+0000","last_clean":"2024-05-09T05:00:18.652587+0000","last_became_active":"2024-05-09T05:00:12.853171+0000","last_became_peered":"2024-05-09T05:00:12.853171+0000","last_unstale":"2024-05-09T05:00:18.652587+0000","last_undegraded":"2024-05-09T05:00:18.652587+0000","last_fullsized":"2024-05-09T05:00:18.652587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:01:08.488591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652158+0000","last_change":"2024-05-09T05:00:16.883712+0000","last_active":"2024-05-09T05:00:18.652158+0000","last_peered":"2024-05-09T05:00:18.652158+0000","last_clean":"2024-05-09T05:00:18.652158+0000","last_became_active":"2024-05-09T05:00:16.883004+0000","last_became_peered":"2024-05-09T05:00:16.883004+0000","last_unstale":"2024-05-09T05:00:18.652158+0000","last_undegraded":"2024-05-09T05:00:18.652158+0000","last_fullsized":"2024-05-09T05:00:18.652158+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:27.371200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652845+0000","last_change":"2024-05-09T05:00:11.839791+0000","last_active":"2024-05-09T05:00:18.652845+0000","last_peered":"2024-05-09T05:00:18.652845+0000","last_clean":"2024-05-09T05:00:18.652845+0000","last_became_active":"2024-05-09T05:00:11.839531+0000","last_became_peered":"2024-05-09T05:00:11.839531+0000","last_unstale":"2024-05-09T05:00:18.652845+0000","last_undegraded":"2024-05-09T05:00:18.652845+0000","last_fullsized":"2024-05-09T05:00:18.652845+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:23.679256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623978+0000","last_change":"2024-05-09T05:00:12.844866+0000","last_active":"2024-05-09T05:00:19.623978+0000","last_peered":"2024-05-09T05:00:19.623978+0000","last_clean":"2024-05-09T05:00:19.623978+0000","last_became_active":"2024-05-09T05:00:12.844015+0000","last_became_peered":"2024-05-09T05:00:12.844015+0000","last_unstale":"2024-05-09T05:00:19.623978+0000","last_undegraded":"2024-05-09T05:00:19.623978+0000","last_fullsized":"2024-05-09T05:00:19.623978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881734+0000","last_change":"2024-05-09T05:00:16.878021+0000","last_active":"2024-05-09T05:00:19.881734+0000","last_peered":"2024-05-09T05:00:19.881734+0000","last_clean":"2024-05-09T05:00:19.881734+0000","last_became_active":"2024-05-09T05:00:16.877322+0000","last_became_peered":"2024-05-09T05:00:16.877322+0000","last_unstale":"2024-05-09T05:00:19.881734+0000","last_undegraded":"2024-05-09T05:00:19.881734+0000","last_fullsized":"2024-05-09T05:00:19.881734+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:34.719408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270583+0000","last_change":"2024-05-09T05:00:11.852994+0000","last_active":"2024-05-09T05:00:20.270583+0000","last_peered":"2024-05-09T05:00:20.270583+0000","last_clean":"2024-05-09T05:00:20.270583+0000","last_became_active":"2024-05-09T05:00:11.852737+0000","last_became_peered":"2024-05-09T05:00:11.852737+0000","last_unstale":"2024-05-09T05:00:20.270583+0000","last_undegraded":"2024-05-09T05:00:20.270583+0000","last_fullsized":"2024-05-09T05:00:20.270583+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:39.794811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970762+0000","last_change":"2024-05-09T05:00:12.845198+0000","last_active":"2024-05-09T05:00:19.970762+0000","last_peered":"2024-05-09T05:00:19.970762+0000","last_clean":"2024-05-09T05:00:19.970762+0000","last_became_active":"2024-05-09T05:00:12.844003+0000","last_became_peered":"2024-05-09T05:00:12.844003+0000","last_unstale":"2024-05-09T05:00:19.970762+0000","last_undegraded":"2024-05-09T05:00:19.970762+0000","last_fullsized":"2024-05-09T05:00:19.970762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:45.314128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882179+0000","last_change":"2024-05-09T05:00:16.877475+0000","last_active":"2024-05-09T05:00:19.882179+0000","last_peered":"2024-05-09T05:00:19.882179+0000","last_clean":"2024-05-09T05:00:19.882179+0000","last_became_active":"2024-05-09T05:00:16.876947+0000","last_became_peered":"2024-05-09T05:00:16.876947+0000","last_unstale":"2024-05-09T05:00:19.882179+0000","last_undegraded":"2024-05-09T05:00:19.882179+0000","last_fullsized":"2024-05-09T05:00:19.882179+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:49.342204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271162+0000","last_change":"2024-05-09T05:00:11.853073+0000","last_active":"2024-05-09T05:00:20.271162+0000","last_peered":"2024-05-09T05:00:20.271162+0000","last_clean":"2024-05-09T05:00:20.271162+0000","last_became_active":"2024-05-09T05:00:11.852759+0000","last_became_peered":"2024-05-09T05:00:11.852759+0000","last_unstale":"2024-05-09T05:00:20.271162+0000","last_undegraded":"2024-05-09T05:00:20.271162+0000","last_fullsized":"2024-05-09T05:00:20.271162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:01.738771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970943+0000","last_change":"2024-05-09T05:00:12.844419+0000","last_active":"2024-05-09T05:00:19.970943+0000","last_peered":"2024-05-09T05:00:19.970943+0000","last_clean":"2024-05-09T05:00:19.970943+0000","last_became_active":"2024-05-09T05:00:12.843951+0000","last_became_peered":"2024-05-09T05:00:12.843951+0000","last_unstale":"2024-05-09T05:00:19.970943+0000","last_undegraded":"2024-05-09T05:00:19.970943+0000","last_fullsized":"2024-05-09T05:00:19.970943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:23.066586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881928+0000","last_change":"2024-05-09T05:00:16.875575+0000","last_active":"2024-05-09T05:00:19.881928+0000","last_peered":"2024-05-09T05:00:19.881928+0000","last_clean":"2024-05-09T05:00:19.881928+0000","last_became_active":"2024-05-09T05:00:16.874993+0000","last_became_peered":"2024-05-09T05:00:16.874993+0000","last_unstale":"2024-05-09T05:00:19.881928+0000","last_undegraded":"2024-05-09T05:00:19.881928+0000","last_fullsized":"2024-05-09T05:00:19.881928+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:34.414100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860116+0000","last_change":"2024-05-09T05:00:12.844310+0000","last_active":"2024-05-09T05:00:17.860116+0000","last_peered":"2024-05-09T05:00:17.860116+0000","last_clean":"2024-05-09T05:00:17.860116+0000","last_became_active":"2024-05-09T05:00:12.843870+0000","last_became_peered":"2024-05-09T05:00:12.843870+0000","last_unstale":"2024-05-09T05:00:17.860116+0000","last_undegraded":"2024-05-09T05:00:17.860116+0000","last_fullsized":"2024-05-09T05:00:17.860116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:19.080163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271207+0000","last_change":"2024-05-09T05:00:11.849654+0000","last_active":"2024-05-09T05:00:20.271207+0000","last_peered":"2024-05-09T05:00:20.271207+0000","last_clean":"2024-05-09T05:00:20.271207+0000","last_became_active":"2024-05-09T05:00:11.849112+0000","last_became_peered":"2024-05-09T05:00:11.849112+0000","last_unstale":"2024-05-09T05:00:20.271207+0000","last_undegraded":"2024-05-09T05:00:20.271207+0000","last_fullsized":"2024-05-09T05:00:20.271207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:04:53.801780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882282+0000","last_change":"2024-05-09T05:00:16.874883+0000","last_active":"2024-05-09T05:00:19.882282+0000","last_peered":"2024-05-09T05:00:19.882282+0000","last_clean":"2024-05-09T05:00:19.882282+0000","last_became_active":"2024-05-09T05:00:16.874390+0000","last_became_peered":"2024-05-09T05:00:16.874390+0000","last_unstale":"2024-05-09T05:00:19.882282+0000","last_undegraded":"2024-05-09T05:00:19.882282+0000","last_fullsized":"2024-05-09T05:00:19.882282+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:09.005203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652613+0000","last_change":"2024-05-09T05:00:12.855440+0000","last_active":"2024-05-09T05:00:18.652613+0000","last_peered":"2024-05-09T05:00:18.652613+0000","last_clean":"2024-05-09T05:00:18.652613+0000","last_became_active":"2024-05-09T05:00:12.854274+0000","last_became_peered":"2024-05-09T05:00:12.854274+0000","last_unstale":"2024-05-09T05:00:18.652613+0000","last_undegraded":"2024-05-09T05:00:18.652613+0000","last_fullsized":"2024-05-09T05:00:18.652613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:30.691332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270354+0000","last_change":"2024-05-09T05:00:11.842122+0000","last_active":"2024-05-09T05:00:20.270354+0000","last_peered":"2024-05-09T05:00:20.270354+0000","last_clean":"2024-05-09T05:00:20.270354+0000","last_became_active":"2024-05-09T05:00:11.840835+0000","last_became_peered":"2024-05-09T05:00:11.840835+0000","last_unstale":"2024-05-09T05:00:20.270354+0000","last_undegraded":"2024-05-09T05:00:20.270354+0000","last_fullsized":"2024-05-09T05:00:20.270354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:41:27.364045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882288+0000","last_change":"2024-05-09T05:00:16.875354+0000","last_active":"2024-05-09T05:00:19.882288+0000","last_peered":"2024-05-09T05:00:19.882288+0000","last_clean":"2024-05-09T05:00:19.882288+0000","last_became_active":"2024-05-09T05:00:16.874942+0000","last_became_peered":"2024-05-09T05:00:16.874942+0000","last_unstale":"2024-05-09T05:00:19.882288+0000","last_undegraded":"2024-05-09T05:00:19.882288+0000","last_fullsized":"2024-05-09T05:00:19.882288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:27.094662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652325+0000","last_change":"2024-05-09T05:00:12.853552+0000","last_active":"2024-05-09T05:00:18.652325+0000","last_peered":"2024-05-09T05:00:18.652325+0000","last_clean":"2024-05-09T05:00:18.652325+0000","last_became_active":"2024-05-09T05:00:12.853268+0000","last_became_peered":"2024-05-09T05:00:12.853268+0000","last_unstale":"2024-05-09T05:00:18.652325+0000","last_undegraded":"2024-05-09T05:00:18.652325+0000","last_fullsized":"2024-05-09T05:00:18.652325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:44.359533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270643+0000","last_change":"2024-05-09T05:00:11.848638+0000","last_active":"2024-05-09T05:00:20.270643+0000","last_peered":"2024-05-09T05:00:20.270643+0000","last_clean":"2024-05-09T05:00:20.270643+0000","last_became_active":"2024-05-09T05:00:11.848351+0000","last_became_peered":"2024-05-09T05:00:11.848351+0000","last_unstale":"2024-05-09T05:00:20.270643+0000","last_undegraded":"2024-05-09T05:00:20.270643+0000","last_fullsized":"2024-05-09T05:00:20.270643+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:01.738771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271321+0000","last_change":"2024-05-09T05:00:16.880744+0000","last_active":"2024-05-09T05:00:20.271321+0000","last_peered":"2024-05-09T05:00:20.271321+0000","last_clean":"2024-05-09T05:00:20.271321+0000","last_became_active":"2024-05-09T05:00:16.880373+0000","last_became_peered":"2024-05-09T05:00:16.880373+0000","last_unstale":"2024-05-09T05:00:20.271321+0000","last_undegraded":"2024-05-09T05:00:20.271321+0000","last_fullsized":"2024-05-09T05:00:20.271321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:10.821888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882019+0000","last_change":"2024-05-09T05:00:12.835034+0000","last_active":"2024-05-09T05:00:19.882019+0000","last_peered":"2024-05-09T05:00:19.882019+0000","last_clean":"2024-05-09T05:00:19.882019+0000","last_became_active":"2024-05-09T05:00:12.834744+0000","last_became_peered":"2024-05-09T05:00:12.834744+0000","last_unstale":"2024-05-09T05:00:19.882019+0000","last_undegraded":"2024-05-09T05:00:19.882019+0000","last_fullsized":"2024-05-09T05:00:19.882019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:26.805203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270527+0000","last_change":"2024-05-09T05:00:11.848682+0000","last_active":"2024-05-09T05:00:20.270527+0000","last_peered":"2024-05-09T05:00:20.270527+0000","last_clean":"2024-05-09T05:00:20.270527+0000","last_became_active":"2024-05-09T05:00:11.848396+0000","last_became_peered":"2024-05-09T05:00:11.848396+0000","last_unstale":"2024-05-09T05:00:20.270527+0000","last_undegraded":"2024-05-09T05:00:20.270527+0000","last_fullsized":"2024-05-09T05:00:20.270527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:44:46.364938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882265+0000","last_change":"2024-05-09T05:00:16.877861+0000","last_active":"2024-05-09T05:00:19.882265+0000","last_peered":"2024-05-09T05:00:19.882265+0000","last_clean":"2024-05-09T05:00:19.882265+0000","last_became_active":"2024-05-09T05:00:16.877155+0000","last_became_peered":"2024-05-09T05:00:16.877155+0000","last_unstale":"2024-05-09T05:00:19.882265+0000","last_undegraded":"2024-05-09T05:00:19.882265+0000","last_fullsized":"2024-05-09T05:00:19.882265+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:01:12.516667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624129+0000","last_change":"2024-05-09T05:00:12.852207+0000","last_active":"2024-05-09T05:00:19.624129+0000","last_peered":"2024-05-09T05:00:19.624129+0000","last_clean":"2024-05-09T05:00:19.624129+0000","last_became_active":"2024-05-09T05:00:12.851927+0000","last_became_peered":"2024-05-09T05:00:12.851927+0000","last_unstale":"2024-05-09T05:00:19.624129+0000","last_undegraded":"2024-05-09T05:00:19.624129+0000","last_fullsized":"2024-05-09T05:00:19.624129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:44.359533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.861099+0000","last_change":"2024-05-09T05:00:11.849008+0000","last_active":"2024-05-09T05:00:17.861099+0000","last_peered":"2024-05-09T05:00:17.861099+0000","last_clean":"2024-05-09T05:00:17.861099+0000","last_became_active":"2024-05-09T05:00:11.848741+0000","last_became_peered":"2024-05-09T05:00:11.848741+0000","last_unstale":"2024-05-09T05:00:17.861099+0000","last_undegraded":"2024-05-09T05:00:17.861099+0000","last_fullsized":"2024-05-09T05:00:17.861099+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:39.794811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881982+0000","last_change":"2024-05-09T05:00:16.879683+0000","last_active":"2024-05-09T05:00:19.881982+0000","last_peered":"2024-05-09T05:00:19.881982+0000","last_clean":"2024-05-09T05:00:19.881982+0000","last_became_active":"2024-05-09T05:00:16.879337+0000","last_became_peered":"2024-05-09T05:00:16.879337+0000","last_unstale":"2024-05-09T05:00:19.881982+0000","last_undegraded":"2024-05-09T05:00:19.881982+0000","last_fullsized":"2024-05-09T05:00:19.881982+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:48.387609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652854+0000","last_change":"2024-05-09T05:00:12.852575+0000","last_active":"2024-05-09T05:00:18.652854+0000","last_peered":"2024-05-09T05:00:18.652854+0000","last_clean":"2024-05-09T05:00:18.652854+0000","last_became_active":"2024-05-09T05:00:12.852308+0000","last_became_peered":"2024-05-09T05:00:12.852308+0000","last_unstale":"2024-05-09T05:00:18.652854+0000","last_undegraded":"2024-05-09T05:00:18.652854+0000","last_fullsized":"2024-05-09T05:00:18.652854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270373+0000","last_change":"2024-05-09T05:00:11.839637+0000","last_active":"2024-05-09T05:00:20.270373+0000","last_peered":"2024-05-09T05:00:20.270373+0000","last_clean":"2024-05-09T05:00:20.270373+0000","last_became_active":"2024-05-09T05:00:11.839214+0000","last_became_peered":"2024-05-09T05:00:11.839214+0000","last_unstale":"2024-05-09T05:00:20.270373+0000","last_undegraded":"2024-05-09T05:00:20.270373+0000","last_fullsized":"2024-05-09T05:00:20.270373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:25.803010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652033+0000","last_change":"2024-05-09T05:00:16.883218+0000","last_active":"2024-05-09T05:00:18.652033+0000","last_peered":"2024-05-09T05:00:18.652033+0000","last_clean":"2024-05-09T05:00:18.652033+0000","last_became_active":"2024-05-09T05:00:16.882353+0000","last_became_peered":"2024-05-09T05:00:16.882353+0000","last_unstale":"2024-05-09T05:00:18.652033+0000","last_undegraded":"2024-05-09T05:00:18.652033+0000","last_fullsized":"2024-05-09T05:00:18.652033+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:02.171370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271136+0000","last_change":"2024-05-09T05:00:12.846656+0000","last_active":"2024-05-09T05:00:20.271136+0000","last_peered":"2024-05-09T05:00:20.271136+0000","last_clean":"2024-05-09T05:00:20.271136+0000","last_became_active":"2024-05-09T05:00:12.846359+0000","last_became_peered":"2024-05-09T05:00:12.846359+0000","last_unstale":"2024-05-09T05:00:20.271136+0000","last_undegraded":"2024-05-09T05:00:20.271136+0000","last_fullsized":"2024-05-09T05:00:20.271136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:49.330395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270574+0000","last_change":"2024-05-09T05:00:11.850170+0000","last_active":"2024-05-09T05:00:20.270574+0000","last_peered":"2024-05-09T05:00:20.270574+0000","last_clean":"2024-05-09T05:00:20.270574+0000","last_became_active":"2024-05-09T05:00:11.849866+0000","last_became_peered":"2024-05-09T05:00:11.849866+0000","last_unstale":"2024-05-09T05:00:20.270574+0000","last_undegraded":"2024-05-09T05:00:20.270574+0000","last_fullsized":"2024-05-09T05:00:20.270574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:27:57.420062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623671+0000","last_change":"2024-05-09T05:00:16.879852+0000","last_active":"2024-05-09T05:00:19.623671+0000","last_peered":"2024-05-09T05:00:19.623671+0000","last_clean":"2024-05-09T05:00:19.623671+0000","last_became_active":"2024-05-09T05:00:16.878219+0000","last_became_peered":"2024-05-09T05:00:16.878219+0000","last_unstale":"2024-05-09T05:00:19.623671+0000","last_undegraded":"2024-05-09T05:00:19.623671+0000","last_fullsized":"2024-05-09T05:00:19.623671+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:20.415639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.971002+0000","last_change":"2024-05-09T05:00:12.859874+0000","last_active":"2024-05-09T05:00:19.971002+0000","last_peered":"2024-05-09T05:00:19.971002+0000","last_clean":"2024-05-09T05:00:19.971002+0000","last_became_active":"2024-05-09T05:00:12.859620+0000","last_became_peered":"2024-05-09T05:00:12.859620+0000","last_unstale":"2024-05-09T05:00:19.971002+0000","last_undegraded":"2024-05-09T05:00:19.971002+0000","last_fullsized":"2024-05-09T05:00:19.971002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:04.977127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270358+0000","last_change":"2024-05-09T05:00:11.838593+0000","last_active":"2024-05-09T05:00:20.270358+0000","last_peered":"2024-05-09T05:00:20.270358+0000","last_clean":"2024-05-09T05:00:20.270358+0000","last_became_active":"2024-05-09T05:00:11.838314+0000","last_became_peered":"2024-05-09T05:00:11.838314+0000","last_unstale":"2024-05-09T05:00:20.270358+0000","last_undegraded":"2024-05-09T05:00:20.270358+0000","last_fullsized":"2024-05-09T05:00:20.270358+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:49.660089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270704+0000","last_change":"2024-05-09T05:00:16.876934+0000","last_active":"2024-05-09T05:00:20.270704+0000","last_peered":"2024-05-09T05:00:20.270704+0000","last_clean":"2024-05-09T05:00:20.270704+0000","last_became_active":"2024-05-09T05:00:16.876531+0000","last_became_peered":"2024-05-09T05:00:16.876531+0000","last_unstale":"2024-05-09T05:00:20.270704+0000","last_undegraded":"2024-05-09T05:00:20.270704+0000","last_fullsized":"2024-05-09T05:00:20.270704+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:54.996456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882435+0000","last_change":"2024-05-09T05:00:12.849957+0000","last_active":"2024-05-09T05:00:19.882435+0000","last_peered":"2024-05-09T05:00:19.882435+0000","last_clean":"2024-05-09T05:00:19.882435+0000","last_became_active":"2024-05-09T05:00:12.849138+0000","last_became_peered":"2024-05-09T05:00:12.849138+0000","last_unstale":"2024-05-09T05:00:19.882435+0000","last_undegraded":"2024-05-09T05:00:19.882435+0000","last_fullsized":"2024-05-09T05:00:19.882435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:02.740965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970665+0000","last_change":"2024-05-09T05:00:11.840734+0000","last_active":"2024-05-09T05:00:19.970665+0000","last_peered":"2024-05-09T05:00:19.970665+0000","last_clean":"2024-05-09T05:00:19.970665+0000","last_became_active":"2024-05-09T05:00:11.840463+0000","last_became_peered":"2024-05-09T05:00:11.840463+0000","last_unstale":"2024-05-09T05:00:19.970665+0000","last_undegraded":"2024-05-09T05:00:19.970665+0000","last_fullsized":"2024-05-09T05:00:19.970665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:49.660089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271060+0000","last_change":"2024-05-09T05:00:16.875120+0000","last_active":"2024-05-09T05:00:20.271060+0000","last_peered":"2024-05-09T05:00:20.271060+0000","last_clean":"2024-05-09T05:00:20.271060+0000","last_became_active":"2024-05-09T05:00:16.874728+0000","last_became_peered":"2024-05-09T05:00:16.874728+0000","last_unstale":"2024-05-09T05:00:20.271060+0000","last_undegraded":"2024-05-09T05:00:20.271060+0000","last_fullsized":"2024-05-09T05:00:20.271060+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:20.415639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271161+0000","last_change":"2024-05-09T05:00:12.851489+0000","last_active":"2024-05-09T05:00:20.271161+0000","last_peered":"2024-05-09T05:00:20.271161+0000","last_clean":"2024-05-09T05:00:20.271161+0000","last_became_active":"2024-05-09T05:00:12.841469+0000","last_became_peered":"2024-05-09T05:00:12.841469+0000","last_unstale":"2024-05-09T05:00:20.271161+0000","last_undegraded":"2024-05-09T05:00:20.271161+0000","last_fullsized":"2024-05-09T05:00:20.271161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:45.463031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860713+0000","last_change":"2024-05-09T05:00:11.841055+0000","last_active":"2024-05-09T05:00:17.860713+0000","last_peered":"2024-05-09T05:00:17.860713+0000","last_clean":"2024-05-09T05:00:17.860713+0000","last_became_active":"2024-05-09T05:00:11.839965+0000","last_became_peered":"2024-05-09T05:00:11.839965+0000","last_unstale":"2024-05-09T05:00:17.860713+0000","last_undegraded":"2024-05-09T05:00:17.860713+0000","last_fullsized":"2024-05-09T05:00:17.860713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:41:27.364045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271386+0000","last_change":"2024-05-09T05:00:16.873524+0000","last_active":"2024-05-09T05:00:20.271386+0000","last_peered":"2024-05-09T05:00:20.271386+0000","last_clean":"2024-05-09T05:00:20.271386+0000","last_became_active":"2024-05-09T05:00:16.872894+0000","last_became_peered":"2024-05-09T05:00:16.872894+0000","last_unstale":"2024-05-09T05:00:20.271386+0000","last_undegraded":"2024-05-09T05:00:20.271386+0000","last_fullsized":"2024-05-09T05:00:20.271386+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:32:00.425543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623506+0000","last_change":"2024-05-09T05:00:12.849280+0000","last_active":"2024-05-09T05:00:19.623506+0000","last_peered":"2024-05-09T05:00:19.623506+0000","last_clean":"2024-05-09T05:00:19.623506+0000","last_became_active":"2024-05-09T05:00:12.848968+0000","last_became_peered":"2024-05-09T05:00:12.848968+0000","last_unstale":"2024-05-09T05:00:19.623506+0000","last_undegraded":"2024-05-09T05:00:19.623506+0000","last_fullsized":"2024-05-09T05:00:19.623506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:49.330395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270849+0000","last_change":"2024-05-09T05:00:11.852903+0000","last_active":"2024-05-09T05:00:20.270849+0000","last_peered":"2024-05-09T05:00:20.270849+0000","last_clean":"2024-05-09T05:00:20.270849+0000","last_became_active":"2024-05-09T05:00:11.852584+0000","last_became_peered":"2024-05-09T05:00:11.852584+0000","last_unstale":"2024-05-09T05:00:20.270849+0000","last_undegraded":"2024-05-09T05:00:20.270849+0000","last_fullsized":"2024-05-09T05:00:20.270849+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:29:13.766278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624019+0000","last_change":"2024-05-09T05:00:16.882378+0000","last_active":"2024-05-09T05:00:19.624019+0000","last_peered":"2024-05-09T05:00:19.624019+0000","last_clean":"2024-05-09T05:00:19.624019+0000","last_became_active":"2024-05-09T05:00:16.881947+0000","last_became_peered":"2024-05-09T05:00:16.881947+0000","last_unstale":"2024-05-09T05:00:19.624019+0000","last_undegraded":"2024-05-09T05:00:19.624019+0000","last_fullsized":"2024-05-09T05:00:19.624019+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:32:00.425543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652370+0000","last_change":"2024-05-09T05:00:12.847632+0000","last_active":"2024-05-09T05:00:18.652370+0000","last_peered":"2024-05-09T05:00:18.652370+0000","last_clean":"2024-05-09T05:00:18.652370+0000","last_became_active":"2024-05-09T05:00:12.847358+0000","last_became_peered":"2024-05-09T05:00:12.847358+0000","last_unstale":"2024-05-09T05:00:18.652370+0000","last_undegraded":"2024-05-09T05:00:18.652370+0000","last_fullsized":"2024-05-09T05:00:18.652370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:49.330395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970011+0000","last_change":"2024-05-09T05:00:11.842747+0000","last_active":"2024-05-09T05:00:19.970011+0000","last_peered":"2024-05-09T05:00:19.970011+0000","last_clean":"2024-05-09T05:00:19.970011+0000","last_became_active":"2024-05-09T05:00:11.842474+0000","last_became_peered":"2024-05-09T05:00:11.842474+0000","last_unstale":"2024-05-09T05:00:19.970011+0000","last_undegraded":"2024-05-09T05:00:19.970011+0000","last_fullsized":"2024-05-09T05:00:19.970011+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:01.738771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":118,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271122+0000","last_change":"2024-05-09T05:00:08.667158+0000","last_active":"2024-05-09T05:00:20.271122+0000","last_peered":"2024-05-09T05:00:20.271122+0000","last_clean":"2024-05-09T05:00:20.271122+0000","last_became_active":"2024-05-09T05:00:08.666316+0000","last_became_peered":"2024-05-09T05:00:08.666316+0000","last_unstale":"2024-05-09T05:00:20.271122+0000","last_undegraded":"2024-05-09T05:00:20.271122+0000","last_fullsized":"2024-05-09T05:00:20.271122+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_clean_scrub_stamp":"2024-05-09T05:00:07.370119+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:20.234643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860739+0000","last_change":"2024-05-09T05:00:16.880367+0000","last_active":"2024-05-09T05:00:17.860739+0000","last_peered":"2024-05-09T05:00:17.860739+0000","last_clean":"2024-05-09T05:00:17.860739+0000","last_became_active":"2024-05-09T05:00:16.879980+0000","last_became_peered":"2024-05-09T05:00:16.879980+0000","last_unstale":"2024-05-09T05:00:17.860739+0000","last_undegraded":"2024-05-09T05:00:17.860739+0000","last_fullsized":"2024-05-09T05:00:17.860739+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:54.996456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271273+0000","last_change":"2024-05-09T05:00:12.852180+0000","last_active":"2024-05-09T05:00:20.271273+0000","last_peered":"2024-05-09T05:00:20.271273+0000","last_clean":"2024-05-09T05:00:20.271273+0000","last_became_active":"2024-05-09T05:00:12.851828+0000","last_became_peered":"2024-05-09T05:00:12.851828+0000","last_unstale":"2024-05-09T05:00:20.271273+0000","last_undegraded":"2024-05-09T05:00:20.271273+0000","last_fullsized":"2024-05-09T05:00:20.271273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:16.387563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.861065+0000","last_change":"2024-05-09T05:00:11.843024+0000","last_active":"2024-05-09T05:00:17.861065+0000","last_peered":"2024-05-09T05:00:17.861065+0000","last_clean":"2024-05-09T05:00:17.861065+0000","last_became_active":"2024-05-09T05:00:11.842703+0000","last_became_peered":"2024-05-09T05:00:11.842703+0000","last_unstale":"2024-05-09T05:00:17.861065+0000","last_undegraded":"2024-05-09T05:00:17.861065+0000","last_fullsized":"2024-05-09T05:00:17.861065+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:48:26.640184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860188+0000","last_change":"2024-05-09T05:00:16.879385+0000","last_active":"2024-05-09T05:00:17.860188+0000","last_peered":"2024-05-09T05:00:17.860188+0000","last_clean":"2024-05-09T05:00:17.860188+0000","last_became_active":"2024-05-09T05:00:16.878956+0000","last_became_peered":"2024-05-09T05:00:16.878956+0000","last_unstale":"2024-05-09T05:00:17.860188+0000","last_undegraded":"2024-05-09T05:00:17.860188+0000","last_fullsized":"2024-05-09T05:00:17.860188+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:49.342204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970802+0000","last_change":"2024-05-09T05:00:12.848110+0000","last_active":"2024-05-09T05:00:19.970802+0000","last_peered":"2024-05-09T05:00:19.970802+0000","last_clean":"2024-05-09T05:00:19.970802+0000","last_became_active":"2024-05-09T05:00:12.847628+0000","last_became_peered":"2024-05-09T05:00:12.847628+0000","last_unstale":"2024-05-09T05:00:19.970802+0000","last_undegraded":"2024-05-09T05:00:19.970802+0000","last_fullsized":"2024-05-09T05:00:19.970802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:16.387563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271019+0000","last_change":"2024-05-09T05:00:11.845441+0000","last_active":"2024-05-09T05:00:20.271019+0000","last_peered":"2024-05-09T05:00:20.271019+0000","last_clean":"2024-05-09T05:00:20.271019+0000","last_became_active":"2024-05-09T05:00:11.845151+0000","last_became_peered":"2024-05-09T05:00:11.845151+0000","last_unstale":"2024-05-09T05:00:20.271019+0000","last_undegraded":"2024-05-09T05:00:20.271019+0000","last_fullsized":"2024-05-09T05:00:20.271019+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:39.794811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623840+0000","last_change":"2024-05-09T05:00:16.880872+0000","last_active":"2024-05-09T05:00:19.623840+0000","last_peered":"2024-05-09T05:00:19.623840+0000","last_clean":"2024-05-09T05:00:19.623840+0000","last_became_active":"2024-05-09T05:00:16.876853+0000","last_became_peered":"2024-05-09T05:00:16.876853+0000","last_unstale":"2024-05-09T05:00:19.623840+0000","last_undegraded":"2024-05-09T05:00:19.623840+0000","last_fullsized":"2024-05-09T05:00:19.623840+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:47:02.206708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271556+0000","last_change":"2024-05-09T05:00:12.854694+0000","last_active":"2024-05-09T05:00:20.271556+0000","last_peered":"2024-05-09T05:00:20.271556+0000","last_clean":"2024-05-09T05:00:20.271556+0000","last_became_active":"2024-05-09T05:00:12.854098+0000","last_became_peered":"2024-05-09T05:00:12.854098+0000","last_unstale":"2024-05-09T05:00:20.271556+0000","last_undegraded":"2024-05-09T05:00:20.271556+0000","last_fullsized":"2024-05-09T05:00:20.271556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:23.343124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860998+0000","last_change":"2024-05-09T05:00:11.835134+0000","last_active":"2024-05-09T05:00:17.860998+0000","last_peered":"2024-05-09T05:00:17.860998+0000","last_clean":"2024-05-09T05:00:17.860998+0000","last_became_active":"2024-05-09T05:00:11.834773+0000","last_became_peered":"2024-05-09T05:00:11.834773+0000","last_unstale":"2024-05-09T05:00:17.860998+0000","last_undegraded":"2024-05-09T05:00:17.860998+0000","last_fullsized":"2024-05-09T05:00:17.860998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:50.824636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970193+0000","last_change":"2024-05-09T05:00:16.879633+0000","last_active":"2024-05-09T05:00:19.970193+0000","last_peered":"2024-05-09T05:00:19.970193+0000","last_clean":"2024-05-09T05:00:19.970193+0000","last_became_active":"2024-05-09T05:00:16.879232+0000","last_became_peered":"2024-05-09T05:00:16.879232+0000","last_unstale":"2024-05-09T05:00:19.970193+0000","last_undegraded":"2024-05-09T05:00:19.970193+0000","last_fullsized":"2024-05-09T05:00:19.970193+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:18:15.413800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271558+0000","last_change":"2024-05-09T05:00:12.852679+0000","last_active":"2024-05-09T05:00:20.271558+0000","last_peered":"2024-05-09T05:00:20.271558+0000","last_clean":"2024-05-09T05:00:20.271558+0000","last_became_active":"2024-05-09T05:00:12.852434+0000","last_became_peered":"2024-05-09T05:00:12.852434+0000","last_unstale":"2024-05-09T05:00:20.271558+0000","last_undegraded":"2024-05-09T05:00:20.271558+0000","last_fullsized":"2024-05-09T05:00:20.271558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:06.793812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881482+0000","last_change":"2024-05-09T05:00:11.845447+0000","last_active":"2024-05-09T05:00:19.881482+0000","last_peered":"2024-05-09T05:00:19.881482+0000","last_clean":"2024-05-09T05:00:19.881482+0000","last_became_active":"2024-05-09T05:00:11.845139+0000","last_became_peered":"2024-05-09T05:00:11.845139+0000","last_unstale":"2024-05-09T05:00:19.881482+0000","last_undegraded":"2024-05-09T05:00:19.881482+0000","last_fullsized":"2024-05-09T05:00:19.881482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:32:22.225189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881692+0000","last_change":"2024-05-09T05:00:16.877859+0000","last_active":"2024-05-09T05:00:19.881692+0000","last_peered":"2024-05-09T05:00:19.881692+0000","last_clean":"2024-05-09T05:00:19.881692+0000","last_became_active":"2024-05-09T05:00:16.877418+0000","last_became_peered":"2024-05-09T05:00:16.877418+0000","last_unstale":"2024-05-09T05:00:19.881692+0000","last_undegraded":"2024-05-09T05:00:19.881692+0000","last_fullsized":"2024-05-09T05:00:19.881692+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:28.924332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271116+0000","last_change":"2024-05-09T05:00:12.855313+0000","last_active":"2024-05-09T05:00:20.271116+0000","last_peered":"2024-05-09T05:00:20.271116+0000","last_clean":"2024-05-09T05:00:20.271116+0000","last_became_active":"2024-05-09T05:00:12.855041+0000","last_became_peered":"2024-05-09T05:00:12.855041+0000","last_unstale":"2024-05-09T05:00:20.271116+0000","last_undegraded":"2024-05-09T05:00:20.271116+0000","last_fullsized":"2024-05-09T05:00:20.271116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:01:08.488591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860579+0000","last_change":"2024-05-09T05:00:11.839443+0000","last_active":"2024-05-09T05:00:17.860579+0000","last_peered":"2024-05-09T05:00:17.860579+0000","last_clean":"2024-05-09T05:00:17.860579+0000","last_became_active":"2024-05-09T05:00:11.839169+0000","last_became_peered":"2024-05-09T05:00:11.839169+0000","last_unstale":"2024-05-09T05:00:17.860579+0000","last_undegraded":"2024-05-09T05:00:17.860579+0000","last_fullsized":"2024-05-09T05:00:17.860579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:10.779395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623875+0000","last_change":"2024-05-09T05:00:16.876207+0000","last_active":"2024-05-09T05:00:19.623875+0000","last_peered":"2024-05-09T05:00:19.623875+0000","last_clean":"2024-05-09T05:00:19.623875+0000","last_became_active":"2024-05-09T05:00:16.875739+0000","last_became_peered":"2024-05-09T05:00:16.875739+0000","last_unstale":"2024-05-09T05:00:19.623875+0000","last_undegraded":"2024-05-09T05:00:19.623875+0000","last_fullsized":"2024-05-09T05:00:19.623875+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:10.821888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860230+0000","last_change":"2024-05-09T05:00:12.851872+0000","last_active":"2024-05-09T05:00:17.860230+0000","last_peered":"2024-05-09T05:00:17.860230+0000","last_clean":"2024-05-09T05:00:17.860230+0000","last_became_active":"2024-05-09T05:00:12.851590+0000","last_became_peered":"2024-05-09T05:00:12.851590+0000","last_unstale":"2024-05-09T05:00:17.860230+0000","last_undegraded":"2024-05-09T05:00:17.860230+0000","last_fullsized":"2024-05-09T05:00:17.860230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:49.330395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860330+0000","last_change":"2024-05-09T05:00:11.845978+0000","last_active":"2024-05-09T05:00:17.860330+0000","last_peered":"2024-05-09T05:00:17.860330+0000","last_clean":"2024-05-09T05:00:17.860330+0000","last_became_active":"2024-05-09T05:00:11.845129+0000","last_became_peered":"2024-05-09T05:00:11.845129+0000","last_unstale":"2024-05-09T05:00:17.860330+0000","last_undegraded":"2024-05-09T05:00:17.860330+0000","last_fullsized":"2024-05-09T05:00:17.860330+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:49.660089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652710+0000","last_change":"2024-05-09T05:00:16.882756+0000","last_active":"2024-05-09T05:00:18.652710+0000","last_peered":"2024-05-09T05:00:18.652710+0000","last_clean":"2024-05-09T05:00:18.652710+0000","last_became_active":"2024-05-09T05:00:16.882378+0000","last_became_peered":"2024-05-09T05:00:16.882378+0000","last_unstale":"2024-05-09T05:00:18.652710+0000","last_undegraded":"2024-05-09T05:00:18.652710+0000","last_fullsized":"2024-05-09T05:00:18.652710+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:51:44.393878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271158+0000","last_change":"2024-05-09T05:00:12.854634+0000","last_active":"2024-05-09T05:00:20.271158+0000","last_peered":"2024-05-09T05:00:20.271158+0000","last_clean":"2024-05-09T05:00:20.271158+0000","last_became_active":"2024-05-09T05:00:12.854366+0000","last_became_peered":"2024-05-09T05:00:12.854366+0000","last_unstale":"2024-05-09T05:00:20.271158+0000","last_undegraded":"2024-05-09T05:00:20.271158+0000","last_fullsized":"2024-05-09T05:00:20.271158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:45.314128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.969990+0000","last_change":"2024-05-09T05:00:11.843045+0000","last_active":"2024-05-09T05:00:19.969990+0000","last_peered":"2024-05-09T05:00:19.969990+0000","last_clean":"2024-05-09T05:00:19.969990+0000","last_became_active":"2024-05-09T05:00:11.842728+0000","last_became_peered":"2024-05-09T05:00:11.842728+0000","last_unstale":"2024-05-09T05:00:19.969990+0000","last_undegraded":"2024-05-09T05:00:19.969990+0000","last_fullsized":"2024-05-09T05:00:19.969990+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:25.803010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652628+0000","last_change":"2024-05-09T05:00:16.880672+0000","last_active":"2024-05-09T05:00:18.652628+0000","last_peered":"2024-05-09T05:00:18.652628+0000","last_clean":"2024-05-09T05:00:18.652628+0000","last_became_active":"2024-05-09T05:00:16.880274+0000","last_became_peered":"2024-05-09T05:00:16.880274+0000","last_unstale":"2024-05-09T05:00:18.652628+0000","last_undegraded":"2024-05-09T05:00:18.652628+0000","last_fullsized":"2024-05-09T05:00:18.652628+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:27:18.295238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623650+0000","last_change":"2024-05-09T05:00:12.854427+0000","last_active":"2024-05-09T05:00:19.623650+0000","last_peered":"2024-05-09T05:00:19.623650+0000","last_clean":"2024-05-09T05:00:19.623650+0000","last_became_active":"2024-05-09T05:00:12.854089+0000","last_became_peered":"2024-05-09T05:00:12.854089+0000","last_unstale":"2024-05-09T05:00:19.623650+0000","last_undegraded":"2024-05-09T05:00:19.623650+0000","last_fullsized":"2024-05-09T05:00:19.623650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:33:12.899897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623926+0000","last_change":"2024-05-09T05:00:11.845606+0000","last_active":"2024-05-09T05:00:19.623926+0000","last_peered":"2024-05-09T05:00:19.623926+0000","last_clean":"2024-05-09T05:00:19.623926+0000","last_became_active":"2024-05-09T05:00:11.845227+0000","last_became_peered":"2024-05-09T05:00:11.845227+0000","last_unstale":"2024-05-09T05:00:19.623926+0000","last_undegraded":"2024-05-09T05:00:19.623926+0000","last_fullsized":"2024-05-09T05:00:19.623926+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:50.824636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970860+0000","last_change":"2024-05-09T05:00:16.880664+0000","last_active":"2024-05-09T05:00:19.970860+0000","last_peered":"2024-05-09T05:00:19.970860+0000","last_clean":"2024-05-09T05:00:19.970860+0000","last_became_active":"2024-05-09T05:00:16.880226+0000","last_became_peered":"2024-05-09T05:00:16.880226+0000","last_unstale":"2024-05-09T05:00:19.970860+0000","last_undegraded":"2024-05-09T05:00:19.970860+0000","last_fullsized":"2024-05-09T05:00:19.970860+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:41.016707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271152+0000","last_change":"2024-05-09T05:00:12.858289+0000","last_active":"2024-05-09T05:00:20.271152+0000","last_peered":"2024-05-09T05:00:20.271152+0000","last_clean":"2024-05-09T05:00:20.271152+0000","last_became_active":"2024-05-09T05:00:12.858013+0000","last_became_peered":"2024-05-09T05:00:12.858013+0000","last_unstale":"2024-05-09T05:00:20.271152+0000","last_undegraded":"2024-05-09T05:00:20.271152+0000","last_fullsized":"2024-05-09T05:00:20.271152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:50.968380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:22.574608+0000","last_change":"2024-05-09T05:00:11.842419+0000","last_active":"2024-05-09T05:00:22.574608+0000","last_peered":"2024-05-09T05:00:22.574608+0000","last_clean":"2024-05-09T05:00:22.574608+0000","last_became_active":"2024-05-09T05:00:11.842112+0000","last_became_peered":"2024-05-09T05:00:11.842112+0000","last_unstale":"2024-05-09T05:00:22.574608+0000","last_undegraded":"2024-05-09T05:00:22.574608+0000","last_fullsized":"2024-05-09T05:00:22.574608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:25.803010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271066+0000","last_change":"2024-05-09T05:00:16.874021+0000","last_active":"2024-05-09T05:00:20.271066+0000","last_peered":"2024-05-09T05:00:20.271066+0000","last_clean":"2024-05-09T05:00:20.271066+0000","last_became_active":"2024-05-09T05:00:16.873587+0000","last_became_peered":"2024-05-09T05:00:16.873587+0000","last_unstale":"2024-05-09T05:00:20.271066+0000","last_undegraded":"2024-05-09T05:00:20.271066+0000","last_fullsized":"2024-05-09T05:00:20.271066+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:47:02.206708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882366+0000","last_change":"2024-05-09T05:00:12.844150+0000","last_active":"2024-05-09T05:00:19.882366+0000","last_peered":"2024-05-09T05:00:19.882366+0000","last_clean":"2024-05-09T05:00:19.882366+0000","last_became_active":"2024-05-09T05:00:12.843845+0000","last_became_peered":"2024-05-09T05:00:12.843845+0000","last_unstale":"2024-05-09T05:00:19.882366+0000","last_undegraded":"2024-05-09T05:00:19.882366+0000","last_fullsized":"2024-05-09T05:00:19.882366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:27:58.422255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270961+0000","last_change":"2024-05-09T05:00:11.842387+0000","last_active":"2024-05-09T05:00:20.270961+0000","last_peered":"2024-05-09T05:00:20.270961+0000","last_clean":"2024-05-09T05:00:20.270961+0000","last_became_active":"2024-05-09T05:00:11.840749+0000","last_became_peered":"2024-05-09T05:00:11.840749+0000","last_unstale":"2024-05-09T05:00:20.270961+0000","last_undegraded":"2024-05-09T05:00:20.270961+0000","last_fullsized":"2024-05-09T05:00:20.270961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:27:57.420062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882161+0000","last_change":"2024-05-09T05:00:16.877382+0000","last_active":"2024-05-09T05:00:19.882161+0000","last_peered":"2024-05-09T05:00:19.882161+0000","last_clean":"2024-05-09T05:00:19.882161+0000","last_became_active":"2024-05-09T05:00:16.876966+0000","last_became_peered":"2024-05-09T05:00:16.876966+0000","last_unstale":"2024-05-09T05:00:19.882161+0000","last_undegraded":"2024-05-09T05:00:19.882161+0000","last_fullsized":"2024-05-09T05:00:19.882161+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:20:41.130506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624095+0000","last_change":"2024-05-09T05:00:12.844211+0000","last_active":"2024-05-09T05:00:19.624095+0000","last_peered":"2024-05-09T05:00:19.624095+0000","last_clean":"2024-05-09T05:00:19.624095+0000","last_became_active":"2024-05-09T05:00:12.843545+0000","last_became_peered":"2024-05-09T05:00:12.843545+0000","last_unstale":"2024-05-09T05:00:19.624095+0000","last_undegraded":"2024-05-09T05:00:19.624095+0000","last_fullsized":"2024-05-09T05:00:19.624095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970032+0000","last_change":"2024-05-09T05:00:11.841156+0000","last_active":"2024-05-09T05:00:19.970032+0000","last_peered":"2024-05-09T05:00:19.970032+0000","last_clean":"2024-05-09T05:00:19.970032+0000","last_became_active":"2024-05-09T05:00:11.840620+0000","last_became_peered":"2024-05-09T05:00:11.840620+0000","last_unstale":"2024-05-09T05:00:19.970032+0000","last_undegraded":"2024-05-09T05:00:19.970032+0000","last_fullsized":"2024-05-09T05:00:19.970032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:10.779395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652218+0000","last_change":"2024-05-09T05:00:16.875531+0000","last_active":"2024-05-09T05:00:18.652218+0000","last_peered":"2024-05-09T05:00:18.652218+0000","last_clean":"2024-05-09T05:00:18.652218+0000","last_became_active":"2024-05-09T05:00:16.875056+0000","last_became_peered":"2024-05-09T05:00:16.875056+0000","last_unstale":"2024-05-09T05:00:18.652218+0000","last_undegraded":"2024-05-09T05:00:18.652218+0000","last_fullsized":"2024-05-09T05:00:18.652218+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:41.016707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970758+0000","last_change":"2024-05-09T05:00:12.852295+0000","last_active":"2024-05-09T05:00:19.970758+0000","last_peered":"2024-05-09T05:00:19.970758+0000","last_clean":"2024-05-09T05:00:19.970758+0000","last_became_active":"2024-05-09T05:00:12.851993+0000","last_became_peered":"2024-05-09T05:00:12.851993+0000","last_unstale":"2024-05-09T05:00:19.970758+0000","last_undegraded":"2024-05-09T05:00:19.970758+0000","last_fullsized":"2024-05-09T05:00:19.970758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:44.359533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860982+0000","last_change":"2024-05-09T05:00:11.840063+0000","last_active":"2024-05-09T05:00:17.860982+0000","last_peered":"2024-05-09T05:00:17.860982+0000","last_clean":"2024-05-09T05:00:17.860982+0000","last_became_active":"2024-05-09T05:00:11.839772+0000","last_became_peered":"2024-05-09T05:00:11.839772+0000","last_unstale":"2024-05-09T05:00:17.860982+0000","last_undegraded":"2024-05-09T05:00:17.860982+0000","last_fullsized":"2024-05-09T05:00:17.860982+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:01.738771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652211+0000","last_change":"2024-05-09T05:00:16.883403+0000","last_active":"2024-05-09T05:00:18.652211+0000","last_peered":"2024-05-09T05:00:18.652211+0000","last_clean":"2024-05-09T05:00:18.652211+0000","last_became_active":"2024-05-09T05:00:16.882440+0000","last_became_peered":"2024-05-09T05:00:16.882440+0000","last_unstale":"2024-05-09T05:00:18.652211+0000","last_undegraded":"2024-05-09T05:00:18.652211+0000","last_fullsized":"2024-05-09T05:00:18.652211+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:18:15.413800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624058+0000","last_change":"2024-05-09T05:00:16.883006+0000","last_active":"2024-05-09T05:00:19.624058+0000","last_peered":"2024-05-09T05:00:19.624058+0000","last_clean":"2024-05-09T05:00:19.624058+0000","last_became_active":"2024-05-09T05:00:16.882624+0000","last_became_peered":"2024-05-09T05:00:16.882624+0000","last_unstale":"2024-05-09T05:00:19.624058+0000","last_undegraded":"2024-05-09T05:00:19.624058+0000","last_fullsized":"2024-05-09T05:00:19.624058+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:02.171370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270485+0000","last_change":"2024-05-09T05:00:12.850411+0000","last_active":"2024-05-09T05:00:20.270485+0000","last_peered":"2024-05-09T05:00:20.270485+0000","last_clean":"2024-05-09T05:00:20.270485+0000","last_became_active":"2024-05-09T05:00:12.850121+0000","last_became_peered":"2024-05-09T05:00:12.850121+0000","last_unstale":"2024-05-09T05:00:20.270485+0000","last_undegraded":"2024-05-09T05:00:20.270485+0000","last_fullsized":"2024-05-09T05:00:20.270485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:30.691332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970418+0000","last_change":"2024-05-09T05:00:11.843746+0000","last_active":"2024-05-09T05:00:19.970418+0000","last_peered":"2024-05-09T05:00:19.970418+0000","last_clean":"2024-05-09T05:00:19.970418+0000","last_became_active":"2024-05-09T05:00:11.843423+0000","last_became_peered":"2024-05-09T05:00:11.843423+0000","last_unstale":"2024-05-09T05:00:19.970418+0000","last_undegraded":"2024-05-09T05:00:19.970418+0000","last_fullsized":"2024-05-09T05:00:19.970418+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:41:27.364045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860975+0000","last_change":"2024-05-09T05:00:16.873412+0000","last_active":"2024-05-09T05:00:17.860975+0000","last_peered":"2024-05-09T05:00:17.860975+0000","last_clean":"2024-05-09T05:00:17.860975+0000","last_became_active":"2024-05-09T05:00:16.873014+0000","last_became_peered":"2024-05-09T05:00:16.873014+0000","last_unstale":"2024-05-09T05:00:17.860975+0000","last_undegraded":"2024-05-09T05:00:17.860975+0000","last_fullsized":"2024-05-09T05:00:17.860975+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:20.415639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652208+0000","last_change":"2024-05-09T05:00:12.856243+0000","last_active":"2024-05-09T05:00:18.652208+0000","last_peered":"2024-05-09T05:00:18.652208+0000","last_clean":"2024-05-09T05:00:18.652208+0000","last_became_active":"2024-05-09T05:00:12.854306+0000","last_became_peered":"2024-05-09T05:00:12.854306+0000","last_unstale":"2024-05-09T05:00:18.652208+0000","last_undegraded":"2024-05-09T05:00:18.652208+0000","last_fullsized":"2024-05-09T05:00:18.652208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:10:10.149613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652448+0000","last_change":"2024-05-09T05:00:11.837046+0000","last_active":"2024-05-09T05:00:18.652448+0000","last_peered":"2024-05-09T05:00:18.652448+0000","last_clean":"2024-05-09T05:00:18.652448+0000","last_became_active":"2024-05-09T05:00:11.836773+0000","last_became_peered":"2024-05-09T05:00:11.836773+0000","last_unstale":"2024-05-09T05:00:18.652448+0000","last_undegraded":"2024-05-09T05:00:18.652448+0000","last_fullsized":"2024-05-09T05:00:18.652448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:48:26.640184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860385+0000","last_change":"2024-05-09T05:00:11.841442+0000","last_active":"2024-05-09T05:00:17.860385+0000","last_peered":"2024-05-09T05:00:17.860385+0000","last_clean":"2024-05-09T05:00:17.860385+0000","last_became_active":"2024-05-09T05:00:11.841099+0000","last_became_peered":"2024-05-09T05:00:11.841099+0000","last_unstale":"2024-05-09T05:00:17.860385+0000","last_undegraded":"2024-05-09T05:00:17.860385+0000","last_fullsized":"2024-05-09T05:00:17.860385+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:41:27.364045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623878+0000","last_change":"2024-05-09T05:00:12.845049+0000","last_active":"2024-05-09T05:00:19.623878+0000","last_peered":"2024-05-09T05:00:19.623878+0000","last_clean":"2024-05-09T05:00:19.623878+0000","last_became_active":"2024-05-09T05:00:12.844724+0000","last_became_peered":"2024-05-09T05:00:12.844724+0000","last_unstale":"2024-05-09T05:00:19.623878+0000","last_undegraded":"2024-05-09T05:00:19.623878+0000","last_fullsized":"2024-05-09T05:00:19.623878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:45.463031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860933+0000","last_change":"2024-05-09T05:00:16.883497+0000","last_active":"2024-05-09T05:00:17.860933+0000","last_peered":"2024-05-09T05:00:17.860933+0000","last_clean":"2024-05-09T05:00:17.860933+0000","last_became_active":"2024-05-09T05:00:16.883063+0000","last_became_peered":"2024-05-09T05:00:16.883063+0000","last_unstale":"2024-05-09T05:00:17.860933+0000","last_undegraded":"2024-05-09T05:00:17.860933+0000","last_fullsized":"2024-05-09T05:00:17.860933+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:04:58.832049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652723+0000","last_change":"2024-05-09T05:00:11.847505+0000","last_active":"2024-05-09T05:00:18.652723+0000","last_peered":"2024-05-09T05:00:18.652723+0000","last_clean":"2024-05-09T05:00:18.652723+0000","last_became_active":"2024-05-09T05:00:11.847187+0000","last_became_peered":"2024-05-09T05:00:11.847187+0000","last_unstale":"2024-05-09T05:00:18.652723+0000","last_undegraded":"2024-05-09T05:00:18.652723+0000","last_fullsized":"2024-05-09T05:00:18.652723+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:50.824636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882122+0000","last_change":"2024-05-09T05:00:12.838998+0000","last_active":"2024-05-09T05:00:19.882122+0000","last_peered":"2024-05-09T05:00:19.882122+0000","last_clean":"2024-05-09T05:00:19.882122+0000","last_became_active":"2024-05-09T05:00:12.838702+0000","last_became_peered":"2024-05-09T05:00:12.838702+0000","last_unstale":"2024-05-09T05:00:19.882122+0000","last_undegraded":"2024-05-09T05:00:19.882122+0000","last_fullsized":"2024-05-09T05:00:19.882122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:50.662283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652939+0000","last_change":"2024-05-09T05:00:16.876805+0000","last_active":"2024-05-09T05:00:18.652939+0000","last_peered":"2024-05-09T05:00:18.652939+0000","last_clean":"2024-05-09T05:00:18.652939+0000","last_became_active":"2024-05-09T05:00:16.876265+0000","last_became_peered":"2024-05-09T05:00:16.876265+0000","last_unstale":"2024-05-09T05:00:18.652939+0000","last_undegraded":"2024-05-09T05:00:18.652939+0000","last_fullsized":"2024-05-09T05:00:18.652939+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:54:27.622037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860099+0000","last_change":"2024-05-09T05:00:12.857527+0000","last_active":"2024-05-09T05:00:17.860099+0000","last_peered":"2024-05-09T05:00:17.860099+0000","last_clean":"2024-05-09T05:00:17.860099+0000","last_became_active":"2024-05-09T05:00:12.857202+0000","last_became_peered":"2024-05-09T05:00:12.857202+0000","last_unstale":"2024-05-09T05:00:17.860099+0000","last_undegraded":"2024-05-09T05:00:17.860099+0000","last_fullsized":"2024-05-09T05:00:17.860099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:45.463031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860631+0000","last_change":"2024-05-09T05:00:12.844334+0000","last_active":"2024-05-09T05:00:17.860631+0000","last_peered":"2024-05-09T05:00:17.860631+0000","last_clean":"2024-05-09T05:00:17.860631+0000","last_became_active":"2024-05-09T05:00:12.844018+0000","last_became_peered":"2024-05-09T05:00:12.844018+0000","last_unstale":"2024-05-09T05:00:17.860631+0000","last_undegraded":"2024-05-09T05:00:17.860631+0000","last_fullsized":"2024-05-09T05:00:17.860631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:40.797004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970591+0000","last_change":"2024-05-09T05:00:12.855985+0000","last_active":"2024-05-09T05:00:19.970591+0000","last_peered":"2024-05-09T05:00:19.970591+0000","last_clean":"2024-05-09T05:00:19.970591+0000","last_became_active":"2024-05-09T05:00:12.855586+0000","last_became_peered":"2024-05-09T05:00:12.855586+0000","last_unstale":"2024-05-09T05:00:19.970591+0000","last_undegraded":"2024-05-09T05:00:19.970591+0000","last_fullsized":"2024-05-09T05:00:19.970591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:06.793812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623781+0000","last_change":"2024-05-09T05:00:12.848323+0000","last_active":"2024-05-09T05:00:19.623781+0000","last_peered":"2024-05-09T05:00:19.623781+0000","last_clean":"2024-05-09T05:00:19.623781+0000","last_became_active":"2024-05-09T05:00:12.848053+0000","last_became_peered":"2024-05-09T05:00:12.848053+0000","last_unstale":"2024-05-09T05:00:19.623781+0000","last_undegraded":"2024-05-09T05:00:19.623781+0000","last_fullsized":"2024-05-09T05:00:19.623781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:19.080163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652749+0000","last_change":"2024-05-09T05:00:12.849804+0000","last_active":"2024-05-09T05:00:18.652749+0000","last_peered":"2024-05-09T05:00:18.652749+0000","last_clean":"2024-05-09T05:00:18.652749+0000","last_became_active":"2024-05-09T05:00:12.849530+0000","last_became_peered":"2024-05-09T05:00:12.849530+0000","last_unstale":"2024-05-09T05:00:18.652749+0000","last_undegraded":"2024-05-09T05:00:18.652749+0000","last_fullsized":"2024-05-09T05:00:18.652749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:20:37.102430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270858+0000","last_change":"2024-05-09T05:00:12.848438+0000","last_active":"2024-05-09T05:00:20.270858+0000","last_peered":"2024-05-09T05:00:20.270858+0000","last_clean":"2024-05-09T05:00:20.270858+0000","last_became_active":"2024-05-09T05:00:12.848176+0000","last_became_peered":"2024-05-09T05:00:12.848176+0000","last_unstale":"2024-05-09T05:00:20.270858+0000","last_undegraded":"2024-05-09T05:00:20.270858+0000","last_fullsized":"2024-05-09T05:00:20.270858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860997+0000","last_change":"2024-05-09T05:00:12.855310+0000","last_active":"2024-05-09T05:00:17.860997+0000","last_peered":"2024-05-09T05:00:17.860997+0000","last_clean":"2024-05-09T05:00:17.860997+0000","last_became_active":"2024-05-09T05:00:12.854031+0000","last_became_peered":"2024-05-09T05:00:12.854031+0000","last_unstale":"2024-05-09T05:00:17.860997+0000","last_undegraded":"2024-05-09T05:00:17.860997+0000","last_fullsized":"2024-05-09T05:00:17.860997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:44.359533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652800+0000","last_change":"2024-05-09T05:00:12.849125+0000","last_active":"2024-05-09T05:00:18.652800+0000","last_peered":"2024-05-09T05:00:18.652800+0000","last_clean":"2024-05-09T05:00:18.652800+0000","last_became_active":"2024-05-09T05:00:12.848852+0000","last_became_peered":"2024-05-09T05:00:12.848852+0000","last_unstale":"2024-05-09T05:00:18.652800+0000","last_undegraded":"2024-05-09T05:00:18.652800+0000","last_fullsized":"2024-05-09T05:00:18.652800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970301+0000","last_change":"2024-05-09T05:00:12.847892+0000","last_active":"2024-05-09T05:00:19.970301+0000","last_peered":"2024-05-09T05:00:19.970301+0000","last_clean":"2024-05-09T05:00:19.970301+0000","last_became_active":"2024-05-09T05:00:12.847051+0000","last_became_peered":"2024-05-09T05:00:12.847051+0000","last_unstale":"2024-05-09T05:00:19.970301+0000","last_undegraded":"2024-05-09T05:00:19.970301+0000","last_fullsized":"2024-05-09T05:00:19.970301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652323+0000","last_change":"2024-05-09T05:00:12.848463+0000","last_active":"2024-05-09T05:00:18.652323+0000","last_peered":"2024-05-09T05:00:18.652323+0000","last_clean":"2024-05-09T05:00:18.652323+0000","last_became_active":"2024-05-09T05:00:12.848170+0000","last_became_peered":"2024-05-09T05:00:12.848170+0000","last_unstale":"2024-05-09T05:00:18.652323+0000","last_undegraded":"2024-05-09T05:00:18.652323+0000","last_fullsized":"2024-05-09T05:00:18.652323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:45.463031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970492+0000","last_change":"2024-05-09T05:00:12.852750+0000","last_active":"2024-05-09T05:00:19.970492+0000","last_peered":"2024-05-09T05:00:19.970492+0000","last_clean":"2024-05-09T05:00:19.970492+0000","last_became_active":"2024-05-09T05:00:12.852487+0000","last_became_peered":"2024-05-09T05:00:12.852487+0000","last_unstale":"2024-05-09T05:00:19.970492+0000","last_undegraded":"2024-05-09T05:00:19.970492+0000","last_fullsized":"2024-05-09T05:00:19.970492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:50.968380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623652+0000","last_change":"2024-05-09T05:00:12.850281+0000","last_active":"2024-05-09T05:00:19.623652+0000","last_peered":"2024-05-09T05:00:19.623652+0000","last_clean":"2024-05-09T05:00:19.623652+0000","last_became_active":"2024-05-09T05:00:12.849875+0000","last_became_peered":"2024-05-09T05:00:12.849875+0000","last_unstale":"2024-05-09T05:00:19.623652+0000","last_undegraded":"2024-05-09T05:00:19.623652+0000","last_fullsized":"2024-05-09T05:00:19.623652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:10:10.149613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860536+0000","last_change":"2024-05-09T05:00:12.841388+0000","last_active":"2024-05-09T05:00:17.860536+0000","last_peered":"2024-05-09T05:00:17.860536+0000","last_clean":"2024-05-09T05:00:17.860536+0000","last_became_active":"2024-05-09T05:00:12.841121+0000","last_became_peered":"2024-05-09T05:00:12.841121+0000","last_unstale":"2024-05-09T05:00:17.860536+0000","last_undegraded":"2024-05-09T05:00:17.860536+0000","last_fullsized":"2024-05-09T05:00:17.860536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:30.691332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271354+0000","last_change":"2024-05-09T05:00:12.853880+0000","last_active":"2024-05-09T05:00:20.271354+0000","last_peered":"2024-05-09T05:00:20.271354+0000","last_clean":"2024-05-09T05:00:20.271354+0000","last_became_active":"2024-05-09T05:00:12.852044+0000","last_became_peered":"2024-05-09T05:00:12.852044+0000","last_unstale":"2024-05-09T05:00:20.271354+0000","last_undegraded":"2024-05-09T05:00:20.271354+0000","last_fullsized":"2024-05-09T05:00:20.271354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:31:56.397467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860942+0000","last_change":"2024-05-09T05:00:12.842477+0000","last_active":"2024-05-09T05:00:17.860942+0000","last_peered":"2024-05-09T05:00:17.860942+0000","last_clean":"2024-05-09T05:00:17.860942+0000","last_became_active":"2024-05-09T05:00:12.842135+0000","last_became_peered":"2024-05-09T05:00:12.842135+0000","last_unstale":"2024-05-09T05:00:17.860942+0000","last_undegraded":"2024-05-09T05:00:17.860942+0000","last_fullsized":"2024-05-09T05:00:17.860942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:27:58.422255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.971029+0000","last_change":"2024-05-09T05:00:12.852559+0000","last_active":"2024-05-09T05:00:19.971029+0000","last_peered":"2024-05-09T05:00:19.971029+0000","last_clean":"2024-05-09T05:00:19.971029+0000","last_became_active":"2024-05-09T05:00:12.852370+0000","last_became_peered":"2024-05-09T05:00:12.852370+0000","last_unstale":"2024-05-09T05:00:19.971029+0000","last_undegraded":"2024-05-09T05:00:19.971029+0000","last_fullsized":"2024-05-09T05:00:19.971029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:58.178632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970409+0000","last_change":"2024-05-09T05:00:12.856404+0000","last_active":"2024-05-09T05:00:19.970409+0000","last_peered":"2024-05-09T05:00:19.970409+0000","last_clean":"2024-05-09T05:00:19.970409+0000","last_became_active":"2024-05-09T05:00:12.856126+0000","last_became_peered":"2024-05-09T05:00:12.856126+0000","last_unstale":"2024-05-09T05:00:19.970409+0000","last_undegraded":"2024-05-09T05:00:19.970409+0000","last_fullsized":"2024-05-09T05:00:19.970409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270574+0000","last_change":"2024-05-09T05:00:12.847697+0000","last_active":"2024-05-09T05:00:20.270574+0000","last_peered":"2024-05-09T05:00:20.270574+0000","last_clean":"2024-05-09T05:00:20.270574+0000","last_became_active":"2024-05-09T05:00:12.847412+0000","last_became_peered":"2024-05-09T05:00:12.847412+0000","last_unstale":"2024-05-09T05:00:20.270574+0000","last_undegraded":"2024-05-09T05:00:20.270574+0000","last_fullsized":"2024-05-09T05:00:20.270574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:19.080163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623651+0000","last_change":"2024-05-09T05:00:12.846578+0000","last_active":"2024-05-09T05:00:19.623651+0000","last_peered":"2024-05-09T05:00:19.623651+0000","last_clean":"2024-05-09T05:00:19.623651+0000","last_became_active":"2024-05-09T05:00:12.844465+0000","last_became_peered":"2024-05-09T05:00:12.844465+0000","last_unstale":"2024-05-09T05:00:19.623651+0000","last_undegraded":"2024-05-09T05:00:19.623651+0000","last_fullsized":"2024-05-09T05:00:19.623651+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:17:11.682946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652503+0000","last_change":"2024-05-09T05:00:12.852179+0000","last_active":"2024-05-09T05:00:18.652503+0000","last_peered":"2024-05-09T05:00:18.652503+0000","last_clean":"2024-05-09T05:00:18.652503+0000","last_became_active":"2024-05-09T05:00:12.849715+0000","last_became_peered":"2024-05-09T05:00:12.849715+0000","last_unstale":"2024-05-09T05:00:18.652503+0000","last_undegraded":"2024-05-09T05:00:18.652503+0000","last_fullsized":"2024-05-09T05:00:18.652503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:19.080163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882233+0000","last_change":"2024-05-09T05:00:12.837696+0000","last_active":"2024-05-09T05:00:19.882233+0000","last_peered":"2024-05-09T05:00:19.882233+0000","last_clean":"2024-05-09T05:00:19.882233+0000","last_became_active":"2024-05-09T05:00:12.837445+0000","last_became_peered":"2024-05-09T05:00:12.837445+0000","last_unstale":"2024-05-09T05:00:19.882233+0000","last_undegraded":"2024-05-09T05:00:19.882233+0000","last_fullsized":"2024-05-09T05:00:19.882233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:40.797004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860739+0000","last_change":"2024-05-09T05:00:12.840693+0000","last_active":"2024-05-09T05:00:17.860739+0000","last_peered":"2024-05-09T05:00:17.860739+0000","last_clean":"2024-05-09T05:00:17.860739+0000","last_became_active":"2024-05-09T05:00:12.840400+0000","last_became_peered":"2024-05-09T05:00:12.840400+0000","last_unstale":"2024-05-09T05:00:17.860739+0000","last_undegraded":"2024-05-09T05:00:17.860739+0000","last_fullsized":"2024-05-09T05:00:17.860739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860995+0000","last_change":"2024-05-09T05:00:12.841318+0000","last_active":"2024-05-09T05:00:17.860995+0000","last_peered":"2024-05-09T05:00:17.860995+0000","last_clean":"2024-05-09T05:00:17.860995+0000","last_became_active":"2024-05-09T05:00:12.841026+0000","last_became_peered":"2024-05-09T05:00:12.841026+0000","last_unstale":"2024-05-09T05:00:17.860995+0000","last_undegraded":"2024-05-09T05:00:17.860995+0000","last_fullsized":"2024-05-09T05:00:17.860995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271552+0000","last_change":"2024-05-09T05:00:12.852840+0000","last_active":"2024-05-09T05:00:20.271552+0000","last_peered":"2024-05-09T05:00:20.271552+0000","last_clean":"2024-05-09T05:00:20.271552+0000","last_became_active":"2024-05-09T05:00:12.852259+0000","last_became_peered":"2024-05-09T05:00:12.852259+0000","last_unstale":"2024-05-09T05:00:20.271552+0000","last_undegraded":"2024-05-09T05:00:20.271552+0000","last_fullsized":"2024-05-09T05:00:20.271552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:16:58.143294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652120+0000","last_change":"2024-05-09T05:00:12.846320+0000","last_active":"2024-05-09T05:00:18.652120+0000","last_peered":"2024-05-09T05:00:18.652120+0000","last_clean":"2024-05-09T05:00:18.652120+0000","last_became_active":"2024-05-09T05:00:12.845732+0000","last_became_peered":"2024-05-09T05:00:12.845732+0000","last_unstale":"2024-05-09T05:00:18.652120+0000","last_undegraded":"2024-05-09T05:00:18.652120+0000","last_fullsized":"2024-05-09T05:00:18.652120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:33:12.899897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270936+0000","last_change":"2024-05-09T05:00:12.858255+0000","last_active":"2024-05-09T05:00:20.270936+0000","last_peered":"2024-05-09T05:00:20.270936+0000","last_clean":"2024-05-09T05:00:20.270936+0000","last_became_active":"2024-05-09T05:00:12.857971+0000","last_became_peered":"2024-05-09T05:00:12.857971+0000","last_unstale":"2024-05-09T05:00:20.270936+0000","last_undegraded":"2024-05-09T05:00:20.270936+0000","last_fullsized":"2024-05-09T05:00:20.270936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:23.066586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970422+0000","last_change":"2024-05-09T05:00:12.846491+0000","last_active":"2024-05-09T05:00:19.970422+0000","last_peered":"2024-05-09T05:00:19.970422+0000","last_clean":"2024-05-09T05:00:19.970422+0000","last_became_active":"2024-05-09T05:00:12.845643+0000","last_became_peered":"2024-05-09T05:00:12.845643+0000","last_unstale":"2024-05-09T05:00:19.970422+0000","last_undegraded":"2024-05-09T05:00:19.970422+0000","last_fullsized":"2024-05-09T05:00:19.970422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:20:37.102430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270963+0000","last_change":"2024-05-09T05:00:12.854820+0000","last_active":"2024-05-09T05:00:20.270963+0000","last_peered":"2024-05-09T05:00:20.270963+0000","last_clean":"2024-05-09T05:00:20.270963+0000","last_became_active":"2024-05-09T05:00:12.853663+0000","last_became_peered":"2024-05-09T05:00:12.853663+0000","last_unstale":"2024-05-09T05:00:20.270963+0000","last_undegraded":"2024-05-09T05:00:20.270963+0000","last_fullsized":"2024-05-09T05:00:20.270963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:04.977127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970662+0000","last_change":"2024-05-09T05:00:12.855723+0000","last_active":"2024-05-09T05:00:19.970662+0000","last_peered":"2024-05-09T05:00:19.970662+0000","last_clean":"2024-05-09T05:00:19.970662+0000","last_became_active":"2024-05-09T05:00:12.855479+0000","last_became_peered":"2024-05-09T05:00:12.855479+0000","last_unstale":"2024-05-09T05:00:19.970662+0000","last_undegraded":"2024-05-09T05:00:19.970662+0000","last_fullsized":"2024-05-09T05:00:19.970662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:01:08.488591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623805+0000","last_change":"2024-05-09T05:00:12.838168+0000","last_active":"2024-05-09T05:00:19.623805+0000","last_peered":"2024-05-09T05:00:19.623805+0000","last_clean":"2024-05-09T05:00:19.623805+0000","last_became_active":"2024-05-09T05:00:12.837892+0000","last_became_peered":"2024-05-09T05:00:12.837892+0000","last_unstale":"2024-05-09T05:00:19.623805+0000","last_undegraded":"2024-05-09T05:00:19.623805+0000","last_fullsized":"2024-05-09T05:00:19.623805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:30.691332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271359+0000","last_change":"2024-05-09T05:00:12.851393+0000","last_active":"2024-05-09T05:00:20.271359+0000","last_peered":"2024-05-09T05:00:20.271359+0000","last_clean":"2024-05-09T05:00:20.271359+0000","last_became_active":"2024-05-09T05:00:12.851057+0000","last_became_peered":"2024-05-09T05:00:12.851057+0000","last_unstale":"2024-05-09T05:00:20.271359+0000","last_undegraded":"2024-05-09T05:00:20.271359+0000","last_fullsized":"2024-05-09T05:00:20.271359+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271442+0000","last_change":"2024-05-09T05:00:12.853084+0000","last_active":"2024-05-09T05:00:20.271442+0000","last_peered":"2024-05-09T05:00:20.271442+0000","last_clean":"2024-05-09T05:00:20.271442+0000","last_became_active":"2024-05-09T05:00:12.852602+0000","last_became_peered":"2024-05-09T05:00:12.852602+0000","last_unstale":"2024-05-09T05:00:20.271442+0000","last_undegraded":"2024-05-09T05:00:20.271442+0000","last_fullsized":"2024-05-09T05:00:20.271442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:58.178632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:00:33.389 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-09T05:00:33.389 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-09T05:00:33.389 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-05-09T05:00:33.389 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-05-09T05:00:33.390 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-09T05:00:33.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.527+0000 7ffbdab4d640 1 Processor -- start 2024-05-09T05:00:33.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.527+0000 7ffbdab4d640 1 -- start start 2024-05-09T05:00:33.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.527+0000 7ffbdab4d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ffbd40b4d30 0x7ffbd4169ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:33.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.527+0000 7ffbdab4d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd415f7a0 con 0x7ffbd415f3c0 2024-05-09T05:00:33.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.527+0000 7ffbdab4d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd409f480 con 0x7ffbd4073fb0 2024-05-09T05:00:33.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.527+0000 7ffbdab4d640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ffbd4071cb0 con 0x7ffbd40b4d30 2024-05-09T05:00:33.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.528+0000 7ffbd88e9640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ffbd415f3c0 0x7ffbd409ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38416/0 (socket says 172.21.15.60:38416) 2024-05-09T05:00:33.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.528+0000 7ffbd88e9640 1 -- 172.21.15.60:0/541633797 learned_addr learned my addr 172.21.15.60:0/541633797 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:33.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.528+0000 7ffbd90ea640 1 --2- 172.21.15.60:0/541633797 >> v2:172.21.15.114:3300/0 conn(0x7ffbd40b4d30 0x7ffbd4169ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:33.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.528+0000 7ffbd37fe640 1 -- 172.21.15.60:0/541633797 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 779850713 0 0) 0x7ffbd415f7a0 con 0x7ffbd415f3c0 2024-05-09T05:00:33.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.528+0000 7ffbd37fe640 1 -- 172.21.15.60:0/541633797 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbc4003520 con 0x7ffbd415f3c0 2024-05-09T05:00:33.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.528+0000 7ffbd90ea640 1 -- 172.21.15.60:0/541633797 >> v1:172.21.15.114:6789/0 conn(0x7ffbd4073fb0 legacy=0x7ffbd40b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.528+0000 7ffbd37fe640 1 -- 172.21.15.60:0/541633797 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1818064977 0 0) 0x7ffbd409f480 con 0x7ffbd4073fb0 2024-05-09T05:00:33.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.528+0000 7ffbd90ea640 1 -- 172.21.15.60:0/541633797 >> v1:172.21.15.60:6789/0 conn(0x7ffbd415f3c0 legacy=0x7ffbd409ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.528+0000 7ffbd90ea640 1 -- 172.21.15.60:0/541633797 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd409f760 con 0x7ffbd40b4d30 2024-05-09T05:00:33.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.529+0000 7ffbd90ea640 1 --2- 172.21.15.60:0/541633797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbd40b4d30 0x7ffbd4169ef0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc009d90 tx=0x7ffbcc02a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e47c74b1bbb44c50 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:33.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.529+0000 7ffbd37fe640 1 -- 172.21.15.60:0/541633797 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc007740 con 0x7ffbd40b4d30 2024-05-09T05:00:33.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.529+0000 7ffbd37fe640 1 -- 172.21.15.60:0/541633797 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbcc0078e0 con 0x7ffbd40b4d30 2024-05-09T05:00:33.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.529+0000 7ffbd37fe640 1 -- 172.21.15.60:0/541633797 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc007cb0 con 0x7ffbd40b4d30 2024-05-09T05:00:33.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.530+0000 7ffbdab4d640 1 -- 172.21.15.60:0/541633797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbd40b4d30 msgr2=0x7ffbd4169ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.530+0000 7ffbdab4d640 1 --2- 172.21.15.60:0/541633797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbd40b4d30 0x7ffbd4169ef0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc009d90 tx=0x7ffbcc02a660 comp rx=0 tx=0).stop 2024-05-09T05:00:33.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.530+0000 7ffbdab4d640 1 -- 172.21.15.60:0/541633797 shutdown_connections 2024-05-09T05:00:33.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.530+0000 7ffbdab4d640 1 --2- 172.21.15.60:0/541633797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbd40b4d30 0x7ffbd4169ef0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.530+0000 7ffbdab4d640 1 -- 172.21.15.60:0/541633797 >> 172.21.15.60:0/541633797 conn(0x7ffbd40d1470 msgr2=0x7ffbd40d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:33.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.530+0000 7ffbdab4d640 1 -- 172.21.15.60:0/541633797 shutdown_connections 2024-05-09T05:00:33.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.530+0000 7ffbdab4d640 1 -- 172.21.15.60:0/541633797 wait complete. 2024-05-09T05:00:33.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.531+0000 7ffbdab4d640 1 Processor -- start 2024-05-09T05:00:33.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.531+0000 7ffbdab4d640 1 -- start start 2024-05-09T05:00:33.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.532+0000 7ffbdab4d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbd4073fb0 0x7ffbd415c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:33.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.532+0000 7ffbdab4d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbd40b4d30 0x7ffbd415d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:33.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.532+0000 7ffbdab4d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbd415f3c0 0x7ffbd415da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:33.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.532+0000 7ffbdab4d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd40732d0 con 0x7ffbd415f3c0 2024-05-09T05:00:33.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.532+0000 7ffbdab4d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd4073150 con 0x7ffbd4073fb0 2024-05-09T05:00:33.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.532+0000 7ffbdab4d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ffbd4073450 con 0x7ffbd40b4d30 2024-05-09T05:00:33.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.532+0000 7ffbd90ea640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbd415f3c0 0x7ffbd415da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:33.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.532+0000 7ffbd90ea640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbd415f3c0 0x7ffbd415da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34782/0 (socket says 172.21.15.60:34782) 2024-05-09T05:00:33.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.532+0000 7ffbd90ea640 1 -- 172.21.15.60:0/2395367077 learned_addr learned my addr 172.21.15.60:0/2395367077 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:33.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.532+0000 7ffbd88e9640 1 --2- 172.21.15.60:0/2395367077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbd4073fb0 0x7ffbd415c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:33.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.532+0000 7ffbd3fff640 1 --2- 172.21.15.60:0/2395367077 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbd40b4d30 0x7ffbd415d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:33.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.532+0000 7ffbd90ea640 1 -- 172.21.15.60:0/2395367077 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbd40b4d30 msgr2=0x7ffbd415d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.532+0000 7ffbd90ea640 1 --2- 172.21.15.60:0/2395367077 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbd40b4d30 0x7ffbd415d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.532+0000 7ffbd90ea640 1 -- 172.21.15.60:0/2395367077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbd4073fb0 msgr2=0x7ffbd415c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.532+0000 7ffbd90ea640 1 --2- 172.21.15.60:0/2395367077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbd4073fb0 0x7ffbd415c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.532+0000 7ffbd90ea640 1 -- 172.21.15.60:0/2395367077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd415cbf0 con 0x7ffbd415f3c0 2024-05-09T05:00:33.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.532+0000 7ffbd3fff640 1 --2- 172.21.15.60:0/2395367077 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbd40b4d30 0x7ffbd415d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:33.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.533+0000 7ffbd90ea640 1 --2- 172.21.15.60:0/2395367077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbd415f3c0 0x7ffbd415da00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc009d90 tx=0x7ffbcc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:33.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.533+0000 7ffbd17fa640 1 -- 172.21.15.60:0/2395367077 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc03f070 con 0x7ffbd415f3c0 2024-05-09T05:00:33.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.533+0000 7ffbd17fa640 1 -- 172.21.15.60:0/2395367077 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbcc031b40 con 0x7ffbd415f3c0 2024-05-09T05:00:33.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.533+0000 7ffbdab4d640 1 -- 172.21.15.60:0/2395367077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbd40af970 con 0x7ffbd415f3c0 2024-05-09T05:00:33.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.533+0000 7ffbd17fa640 1 -- 172.21.15.60:0/2395367077 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc043410 con 0x7ffbd415f3c0 2024-05-09T05:00:33.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.533+0000 7ffbdab4d640 1 -- 172.21.15.60:0/2395367077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbd40b0550 con 0x7ffbd415f3c0 2024-05-09T05:00:33.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.534+0000 7ffbdab4d640 1 -- 172.21.15.60:0/2395367077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbd4051040 con 0x7ffbd415f3c0 2024-05-09T05:00:33.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.535+0000 7ffbd17fa640 1 -- 172.21.15.60:0/2395367077 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ffbcc0435b0 con 0x7ffbd415f3c0 2024-05-09T05:00:33.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.535+0000 7ffbd17fa640 1 --2- 172.21.15.60:0/2395367077 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ffbb80762d0 0x7ffbb8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:33.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.535+0000 7ffbd17fa640 1 -- 172.21.15.60:0/2395367077 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ffbcc0b8c40 con 0x7ffbd415f3c0 2024-05-09T05:00:33.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.535+0000 7ffbd88e9640 1 --2- 172.21.15.60:0/2395367077 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ffbb80762d0 0x7ffbb8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:33.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.536+0000 7ffbd88e9640 1 --2- 172.21.15.60:0/2395367077 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ffbb80762d0 0x7ffbb8078790 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8007a60 tx=0x7ffbc8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:33.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.539+0000 7ffbd17fa640 1 -- 172.21.15.60:0/2395367077 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ffbcc0822f0 con 0x7ffbd415f3c0 2024-05-09T05:00:33.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.784+0000 7ffbdab4d640 1 -- 172.21.15.60:0/2395367077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7ffbd4070240 con 0x7ffbd415f3c0 2024-05-09T05:00:33.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.787+0000 7ffbd17fa640 1 -- 172.21.15.60:0/2395367077 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+189528 (secure 0 0 0) 0x7ffbcc04be30 con 0x7ffbd415f3c0 2024-05-09T05:00:33.790 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:33.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.789+0000 7ffbdab4d640 1 -- 172.21.15.60:0/2395367077 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ffbb80762d0 msgr2=0x7ffbb8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.789+0000 7ffbdab4d640 1 --2- 172.21.15.60:0/2395367077 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ffbb80762d0 0x7ffbb8078790 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8007a60 tx=0x7ffbc8009040 comp rx=0 tx=0).stop 2024-05-09T05:00:33.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.789+0000 7ffbdab4d640 1 -- 172.21.15.60:0/2395367077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbd415f3c0 msgr2=0x7ffbd415da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.789+0000 7ffbdab4d640 1 --2- 172.21.15.60:0/2395367077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbd415f3c0 0x7ffbd415da00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc009d90 tx=0x7ffbcc004a40 comp rx=0 tx=0).stop 2024-05-09T05:00:33.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.790+0000 7ffbdab4d640 1 -- 172.21.15.60:0/2395367077 shutdown_connections 2024-05-09T05:00:33.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.790+0000 7ffbdab4d640 1 --2- 172.21.15.60:0/2395367077 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ffbb80762d0 0x7ffbb8078790 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.790+0000 7ffbdab4d640 1 --2- 172.21.15.60:0/2395367077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbd415f3c0 0x7ffbd415da00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.790+0000 7ffbdab4d640 1 --2- 172.21.15.60:0/2395367077 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbd40b4d30 0x7ffbd415d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.790+0000 7ffbdab4d640 1 --2- 172.21.15.60:0/2395367077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbd4073fb0 0x7ffbd415c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.790+0000 7ffbdab4d640 1 -- 172.21.15.60:0/2395367077 >> 172.21.15.60:0/2395367077 conn(0x7ffbd40d1470 msgr2=0x7ffbd4099410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:33.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.790+0000 7ffbdab4d640 1 -- 172.21.15.60:0/2395367077 shutdown_connections 2024-05-09T05:00:33.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.790+0000 7ffbdab4d640 1 -- 172.21.15.60:0/2395367077 wait complete. 2024-05-09T05:00:33.801 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":4075088736},{"type":"v1","addr":"172.21.15.114:6833","nonce":4075088736}]},"active_addr":"172.21.15.114:6833/4075088736","active_change":"2024-05-09T05:00:00.899722+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":3366920860}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":2560763926}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":1180810111}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":4066438235}]}]} 2024-05-09T05:00:33.805 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-05-09T05:00:33.805 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-05-09T05:00:33.805 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:00:33.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.909+0000 7f58d4072640 1 Processor -- start 2024-05-09T05:00:33.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.909+0000 7f58d4072640 1 -- start start 2024-05-09T05:00:33.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.910+0000 7f58d4072640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58cc06f9f0 0x7f58cc06ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:33.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.910+0000 7f58d4072640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58cc06ff60 con 0x7f58cc1606f0 2024-05-09T05:00:33.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.910+0000 7f58d4072640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58cc070240 con 0x7f58cc15f300 2024-05-09T05:00:33.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.910+0000 7f58d4072640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f58cc070520 con 0x7f58cc06f9f0 2024-05-09T05:00:33.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.910+0000 7f58d160d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f58cc1606f0 0x7f58cc0d33f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38420/0 (socket says 172.21.15.60:38420) 2024-05-09T05:00:33.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.910+0000 7f58d160d640 1 -- 172.21.15.60:0/3414997752 learned_addr learned my addr 172.21.15.60:0/3414997752 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:33.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.910+0000 7f58d260f640 1 --2- 172.21.15.60:0/3414997752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58cc06f9f0 0x7f58cc06ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:33.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.910+0000 7f58d0e0c640 1 -- 172.21.15.60:0/3414997752 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1694179518 0 0) 0x7f58cc06ff60 con 0x7f58cc1606f0 2024-05-09T05:00:33.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.910+0000 7f58d0e0c640 1 -- 172.21.15.60:0/3414997752 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58b8003520 con 0x7f58cc1606f0 2024-05-09T05:00:33.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.910+0000 7f58d0e0c640 1 -- 172.21.15.60:0/3414997752 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1527416498 0 0) 0x7f58cc070240 con 0x7f58cc15f300 2024-05-09T05:00:33.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.910+0000 7f58d0e0c640 1 -- 172.21.15.60:0/3414997752 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58cc06ff60 con 0x7f58cc15f300 2024-05-09T05:00:33.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.910+0000 7f58d0e0c640 1 -- 172.21.15.60:0/3414997752 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2555081396 0 0) 0x7f58b8003520 con 0x7f58cc1606f0 2024-05-09T05:00:33.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d0e0c640 1 -- 172.21.15.60:0/3414997752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58cc06f9f0 msgr2=0x7f58cc06ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:33.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d0e0c640 1 --2- 172.21.15.60:0/3414997752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58cc06f9f0 0x7f58cc06ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d0e0c640 1 -- 172.21.15.60:0/3414997752 >> v1:172.21.15.114:6789/0 conn(0x7f58cc15f300 legacy=0x7f58cc1639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d0e0c640 1 -- 172.21.15.60:0/3414997752 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58cc06e740 con 0x7f58cc1606f0 2024-05-09T05:00:33.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d0e0c640 1 -- 172.21.15.60:0/3414997752 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f58bc0016f0 con 0x7f58cc1606f0 2024-05-09T05:00:33.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d260f640 1 --2- 172.21.15.60:0/3414997752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58cc06f9f0 0x7f58cc06ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:00:33.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d0e0c640 1 -- 172.21.15.60:0/3414997752 >> v1:172.21.15.60:6789/0 conn(0x7f58cc1606f0 legacy=0x7f58cc0d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d0e0c640 1 --2- 172.21.15.60:0/3414997752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58b8003c60 0x7f58b8004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:33.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d0e0c640 1 --2- 172.21.15.60:0/3414997752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58b8004800 0x7f58b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:33.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d0e0c640 1 --2- 172.21.15.60:0/3414997752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58cc06f9f0 0x7f58b8005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:33.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d0e0c640 1 -- 172.21.15.60:0/3414997752 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc070520 con 0x7f58b8003c60 2024-05-09T05:00:33.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d0e0c640 1 -- 172.21.15.60:0/3414997752 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f58b800bb50 con 0x7f58b8004800 2024-05-09T05:00:33.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d0e0c640 1 -- 172.21.15.60:0/3414997752 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f58b800bd50 con 0x7f58cc06f9f0 2024-05-09T05:00:33.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d1e0e640 1 --2- 172.21.15.60:0/3414997752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58b8003c60 0x7f58b8004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:33.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d160d640 1 --2- 172.21.15.60:0/3414997752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58b8004800 0x7f58b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:33.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d260f640 1 --2- 172.21.15.60:0/3414997752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58cc06f9f0 0x7f58b8005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:33.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d1e0e640 1 -- 172.21.15.60:0/3414997752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58cc06f9f0 msgr2=0x7f58b8005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:33.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d1e0e640 1 --2- 172.21.15.60:0/3414997752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58cc06f9f0 0x7f58b8005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d1e0e640 1 -- 172.21.15.60:0/3414997752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58b8004800 msgr2=0x7f58b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:33.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d1e0e640 1 --2- 172.21.15.60:0/3414997752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58b8004800 0x7f58b8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.911+0000 7f58d1e0e640 1 -- 172.21.15.60:0/3414997752 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58b8003520 con 0x7f58b8003c60 2024-05-09T05:00:33.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.912+0000 7f58d1e0e640 1 --2- 172.21.15.60:0/3414997752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58b8003c60 0x7f58b8004060 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f58c8001900 tx=0x7f58b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e300b337f9ed312c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:33.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.912+0000 7f58d0e0c640 1 -- 172.21.15.60:0/3414997752 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c8001520 con 0x7f58b8003c60 2024-05-09T05:00:33.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.912+0000 7f58d0e0c640 1 -- 172.21.15.60:0/3414997752 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58c802acd0 con 0x7f58b8003c60 2024-05-09T05:00:33.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.912+0000 7f58d0e0c640 1 -- 172.21.15.60:0/3414997752 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c8005310 con 0x7f58b8003c60 2024-05-09T05:00:33.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.912+0000 7f58d4072640 1 -- 172.21.15.60:0/3414997752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58b8003c60 msgr2=0x7f58b8004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.912+0000 7f58d4072640 1 --2- 172.21.15.60:0/3414997752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58b8003c60 0x7f58b8004060 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f58c8001900 tx=0x7f58b8005830 comp rx=0 tx=0).stop 2024-05-09T05:00:33.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.912+0000 7f58d260f640 1 -- 172.21.15.60:0/3414997752 reap_dead start 2024-05-09T05:00:33.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.912+0000 7f58d4072640 1 -- 172.21.15.60:0/3414997752 shutdown_connections 2024-05-09T05:00:33.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.912+0000 7f58d4072640 1 -- 172.21.15.60:0/3414997752 >> 172.21.15.60:0/3414997752 conn(0x7f58cc09b850 msgr2=0x7f58cc09bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:33.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.913+0000 7f58d4072640 1 -- 172.21.15.60:0/3414997752 shutdown_connections 2024-05-09T05:00:33.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.913+0000 7f58d4072640 1 -- 172.21.15.60:0/3414997752 wait complete. 2024-05-09T05:00:33.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.914+0000 7f58d4072640 1 Processor -- start 2024-05-09T05:00:33.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.914+0000 7f58d4072640 1 -- start start 2024-05-09T05:00:33.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.914+0000 7f58d4072640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b8003c60 0x7f58cc0ac760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:33.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.914+0000 7f58d4072640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58cc15d480 0x7f58cc0ae890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:33.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.914+0000 7f58d4072640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58cc159e00 0x7f58cc0aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:33.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.914+0000 7f58d4072640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc0d3c80 con 0x7f58cc15d480 2024-05-09T05:00:33.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.914+0000 7f58d4072640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f58cc0d3b00 con 0x7f58cc159e00 2024-05-09T05:00:33.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.914+0000 7f58d4072640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f58cc06ef40 con 0x7f58b8003c60 2024-05-09T05:00:33.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.915+0000 7f58d160d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58cc15d480 0x7f58cc0ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:33.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.915+0000 7f58d160d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58cc15d480 0x7f58cc0ae890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34794/0 (socket says 172.21.15.60:34794) 2024-05-09T05:00:33.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.915+0000 7f58d160d640 1 -- 172.21.15.60:0/3067590649 learned_addr learned my addr 172.21.15.60:0/3067590649 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:33.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.915+0000 7f58d1e0e640 1 --2- 172.21.15.60:0/3067590649 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b8003c60 0x7f58cc0ac760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:33.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.915+0000 7f58d260f640 1 --2- 172.21.15.60:0/3067590649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58cc159e00 0x7f58cc0aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:33.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.915+0000 7f58d160d640 1 -- 172.21.15.60:0/3067590649 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b8003c60 msgr2=0x7f58cc0ac760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.915+0000 7f58d160d640 1 --2- 172.21.15.60:0/3067590649 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b8003c60 0x7f58cc0ac760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.915+0000 7f58d160d640 1 -- 172.21.15.60:0/3067590649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58cc159e00 msgr2=0x7f58cc0aedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.915+0000 7f58d160d640 1 --2- 172.21.15.60:0/3067590649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58cc159e00 0x7f58cc0aedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.915+0000 7f58d160d640 1 -- 172.21.15.60:0/3067590649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58cc0adef0 con 0x7f58cc15d480 2024-05-09T05:00:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.915+0000 7f58d160d640 1 --2- 172.21.15.60:0/3067590649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58cc15d480 0x7f58cc0ae890 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f58bc004ef0 tx=0x7f58bc007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.915+0000 7f58c27fc640 1 -- 172.21.15.60:0/3067590649 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58bc00f7b0 con 0x7f58cc15d480 2024-05-09T05:00:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.915+0000 7f58c27fc640 1 -- 172.21.15.60:0/3067590649 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58bc018070 con 0x7f58cc15d480 2024-05-09T05:00:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.915+0000 7f58d4072640 1 -- 172.21.15.60:0/3067590649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58cc0ae1d0 con 0x7f58cc15d480 2024-05-09T05:00:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.915+0000 7f58c27fc640 1 -- 172.21.15.60:0/3067590649 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58bc00fb50 con 0x7f58cc15d480 2024-05-09T05:00:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.915+0000 7f58d4072640 1 -- 172.21.15.60:0/3067590649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58cc0acca0 con 0x7f58cc15d480 2024-05-09T05:00:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.916+0000 7f58d4072640 1 -- 172.21.15.60:0/3067590649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58cc158010 con 0x7f58cc15d480 2024-05-09T05:00:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.917+0000 7f58c27fc640 1 -- 172.21.15.60:0/3067590649 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f58bc00fcf0 con 0x7f58cc15d480 2024-05-09T05:00:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.918+0000 7f58c27fc640 1 --2- 172.21.15.60:0/3067590649 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f58b00762d0 0x7f58b0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:33.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.918+0000 7f58c27fc640 1 -- 172.21.15.60:0/3067590649 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f58bc097950 con 0x7f58cc15d480 2024-05-09T05:00:33.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.918+0000 7f58d1e0e640 1 --2- 172.21.15.60:0/3067590649 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f58b00762d0 0x7f58b0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:33.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.918+0000 7f58d1e0e640 1 --2- 172.21.15.60:0/3067590649 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f58b00762d0 0x7f58b0078790 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f58c8031000 tx=0x7f58c8036040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:33.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:33.922+0000 7f58c27fc640 1 -- 172.21.15.60:0/3067590649 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f58bc060ed0 con 0x7f58cc15d480 2024-05-09T05:00:34.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.114+0000 7f58d4072640 1 -- 172.21.15.60:0/3067590649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f58cc051040 con 0x7f58cc15d480 2024-05-09T05:00:34.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.115+0000 7f58c27fc640 1 -- 172.21.15.60:0/3067590649 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18490 (secure 0 0 0) 0x7f58bc0607d0 con 0x7f58cc15d480 2024-05-09T05:00:34.116 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:34.116 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:00:17.248958+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:00:10.358133+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:00:11.157769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T05:00:14.850459+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:00:34.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.117+0000 7f58d4072640 1 -- 172.21.15.60:0/3067590649 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f58b00762d0 msgr2=0x7f58b0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.117+0000 7f58d4072640 1 --2- 172.21.15.60:0/3067590649 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f58b00762d0 0x7f58b0078790 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f58c8031000 tx=0x7f58c8036040 comp rx=0 tx=0).stop 2024-05-09T05:00:34.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.117+0000 7f58d4072640 1 -- 172.21.15.60:0/3067590649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58cc15d480 msgr2=0x7f58cc0ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.117+0000 7f58d4072640 1 --2- 172.21.15.60:0/3067590649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58cc15d480 0x7f58cc0ae890 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f58bc004ef0 tx=0x7f58bc007430 comp rx=0 tx=0).stop 2024-05-09T05:00:34.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.117+0000 7f58d4072640 1 -- 172.21.15.60:0/3067590649 shutdown_connections 2024-05-09T05:00:34.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.117+0000 7f58d4072640 1 --2- 172.21.15.60:0/3067590649 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f58b00762d0 0x7f58b0078790 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.117+0000 7f58d4072640 1 --2- 172.21.15.60:0/3067590649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58cc159e00 0x7f58cc0aedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.117+0000 7f58d4072640 1 --2- 172.21.15.60:0/3067590649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58cc15d480 0x7f58cc0ae890 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.117+0000 7f58d4072640 1 --2- 172.21.15.60:0/3067590649 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f58b8003c60 0x7f58cc0ac760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.117+0000 7f58d4072640 1 -- 172.21.15.60:0/3067590649 >> 172.21.15.60:0/3067590649 conn(0x7f58cc09b850 msgr2=0x7f58cc073160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:34.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.117+0000 7f58d4072640 1 -- 172.21.15.60:0/3067590649 shutdown_connections 2024-05-09T05:00:34.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.117+0000 7f58d4072640 1 -- 172.21.15.60:0/3067590649 wait complete. 2024-05-09T05:00:34.130 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-05-09T05:00:34.130 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:00:34.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.233+0000 7f4fd7de4640 1 Processor -- start 2024-05-09T05:00:34.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.233+0000 7f4fd7de4640 1 -- start start 2024-05-09T05:00:34.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.234+0000 7f4fd7de4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fd0073fb0 0x7f4fd00b4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.234+0000 7f4fd7de4640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4fd00b4b60 0x7f4fd0072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.234+0000 7f4fd7de4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fd015f7a0 con 0x7f4fd015f3c0 2024-05-09T05:00:34.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.234+0000 7f4fd7de4640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f4fd0071cb0 con 0x7f4fd00b4b60 2024-05-09T05:00:34.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.234+0000 7f4fd7de4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4fd009f480 con 0x7f4fd0073fb0 2024-05-09T05:00:34.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.234+0000 7f4fd5b80640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4fd015f3c0 0x7f4fd009ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38426/0 (socket says 172.21.15.60:38426) 2024-05-09T05:00:34.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.234+0000 7f4fd5b80640 1 -- 172.21.15.60:0/626286059 learned_addr learned my addr 172.21.15.60:0/626286059 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:34.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.234+0000 7f4fd4b7e640 1 -- 172.21.15.60:0/626286059 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 644102173 0 0) 0x7f4fd015f7a0 con 0x7f4fd015f3c0 2024-05-09T05:00:34.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.234+0000 7f4fd4b7e640 1 -- 172.21.15.60:0/626286059 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4fb8003520 con 0x7f4fd015f3c0 2024-05-09T05:00:34.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.234+0000 7f4fd6381640 1 --2- 172.21.15.60:0/626286059 >> v2:172.21.15.114:3300/0 conn(0x7f4fd00b4b60 0x7f4fd0072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.234+0000 7f4fd537f640 1 --2- 172.21.15.60:0/626286059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fd0073fb0 0x7f4fd00b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd4b7e640 1 -- 172.21.15.60:0/626286059 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2995855197 0 0) 0x7f4fb8003520 con 0x7f4fd015f3c0 2024-05-09T05:00:34.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd4b7e640 1 -- 172.21.15.60:0/626286059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fd0073fb0 msgr2=0x7f4fd00b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:34.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd4b7e640 1 --2- 172.21.15.60:0/626286059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fd0073fb0 0x7f4fd00b4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd4b7e640 1 -- 172.21.15.60:0/626286059 >> v2:172.21.15.114:3300/0 conn(0x7f4fd00b4b60 msgr2=0x7f4fd0072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:34.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd4b7e640 1 --2- 172.21.15.60:0/626286059 >> v2:172.21.15.114:3300/0 conn(0x7f4fd00b4b60 0x7f4fd0072140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd4b7e640 1 -- 172.21.15.60:0/626286059 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd009f760 con 0x7f4fd015f3c0 2024-05-09T05:00:34.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd4b7e640 1 -- 172.21.15.60:0/626286059 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f4fc8001a70 con 0x7f4fd015f3c0 2024-05-09T05:00:34.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd4b7e640 1 -- 172.21.15.60:0/626286059 >> v1:172.21.15.60:6789/0 conn(0x7f4fd015f3c0 legacy=0x7f4fd009ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd4b7e640 1 --2- 172.21.15.60:0/626286059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fb8003a00 0x7f4fb8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd4b7e640 1 --2- 172.21.15.60:0/626286059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fb8004800 0x7f4fb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd4b7e640 1 --2- 172.21.15.60:0/626286059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fd0073fb0 0x7f4fb80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd4b7e640 1 -- 172.21.15.60:0/626286059 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd0071cb0 con 0x7f4fb8004800 2024-05-09T05:00:34.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd4b7e640 1 -- 172.21.15.60:0/626286059 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd009f480 con 0x7f4fb8003a00 2024-05-09T05:00:34.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd4b7e640 1 -- 172.21.15.60:0/626286059 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4fb800baf0 con 0x7f4fd0073fb0 2024-05-09T05:00:34.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd537f640 1 --2- 172.21.15.60:0/626286059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fb8004800 0x7f4fb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd5b80640 1 --2- 172.21.15.60:0/626286059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fb8003a00 0x7f4fb8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd6381640 1 --2- 172.21.15.60:0/626286059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fd0073fb0 0x7f4fb80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd537f640 1 -- 172.21.15.60:0/626286059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fd0073fb0 msgr2=0x7f4fb80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd537f640 1 --2- 172.21.15.60:0/626286059 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fd0073fb0 0x7f4fb80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd537f640 1 -- 172.21.15.60:0/626286059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fb8003a00 msgr2=0x7f4fb8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd537f640 1 --2- 172.21.15.60:0/626286059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fb8003a00 0x7f4fb8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.235+0000 7f4fd537f640 1 -- 172.21.15.60:0/626286059 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb8003520 con 0x7f4fb8004800 2024-05-09T05:00:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.236+0000 7f4fd537f640 1 --2- 172.21.15.60:0/626286059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fb8004800 0x7f4fb8004c80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4fc400ba30 tx=0x7f4fb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63cd4f3668e33639 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.236+0000 7f4fd4b7e640 1 -- 172.21.15.60:0/626286059 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4030070 con 0x7f4fb8004800 2024-05-09T05:00:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.236+0000 7f4fd4b7e640 1 -- 172.21.15.60:0/626286059 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fc402ade0 con 0x7f4fb8004800 2024-05-09T05:00:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.236+0000 7f4fd4b7e640 1 -- 172.21.15.60:0/626286059 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc400b600 con 0x7f4fb8004800 2024-05-09T05:00:34.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.236+0000 7f4fd7de4640 1 -- 172.21.15.60:0/626286059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fb8004800 msgr2=0x7f4fb8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.236+0000 7f4fd7de4640 1 --2- 172.21.15.60:0/626286059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fb8004800 0x7f4fb8004c80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4fc400ba30 tx=0x7f4fb8005830 comp rx=0 tx=0).stop 2024-05-09T05:00:34.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.236+0000 7f4fd6381640 1 -- 172.21.15.60:0/626286059 reap_dead start 2024-05-09T05:00:34.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.237+0000 7f4fd7de4640 1 -- 172.21.15.60:0/626286059 shutdown_connections 2024-05-09T05:00:34.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.237+0000 7f4fd7de4640 1 -- 172.21.15.60:0/626286059 >> 172.21.15.60:0/626286059 conn(0x7f4fd00d1470 msgr2=0x7f4fd00d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:34.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.237+0000 7f4fd7de4640 1 -- 172.21.15.60:0/626286059 shutdown_connections 2024-05-09T05:00:34.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.237+0000 7f4fd7de4640 1 -- 172.21.15.60:0/626286059 wait complete. 2024-05-09T05:00:34.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.238+0000 7f4fd7de4640 1 Processor -- start 2024-05-09T05:00:34.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.238+0000 7f4fd7de4640 1 -- start start 2024-05-09T05:00:34.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fd7de4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fb8004800 0x7f4fd01f5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fd7de4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fd00ac6e0 0x7f4fd0161670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fd7de4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fd01593d0 0x7f4fd015c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fd7de4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd00b47a0 con 0x7f4fd00ac6e0 2024-05-09T05:00:34.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fd7de4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd00b4620 con 0x7f4fd01593d0 2024-05-09T05:00:34.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fd7de4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4fd00b4920 con 0x7f4fb8004800 2024-05-09T05:00:34.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fd537f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fd00ac6e0 0x7f4fd0161670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fd537f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fd00ac6e0 0x7f4fd0161670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34814/0 (socket says 172.21.15.60:34814) 2024-05-09T05:00:34.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fd537f640 1 -- 172.21.15.60:0/2762912521 learned_addr learned my addr 172.21.15.60:0/2762912521 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:34.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fd5b80640 1 --2- 172.21.15.60:0/2762912521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fb8004800 0x7f4fd01f5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fd6381640 1 --2- 172.21.15.60:0/2762912521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fd01593d0 0x7f4fd015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fd537f640 1 -- 172.21.15.60:0/2762912521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fb8004800 msgr2=0x7f4fd01f5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fd537f640 1 --2- 172.21.15.60:0/2762912521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fb8004800 0x7f4fd01f5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fd537f640 1 -- 172.21.15.60:0/2762912521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fd01593d0 msgr2=0x7f4fd015c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fd537f640 1 --2- 172.21.15.60:0/2762912521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fd01593d0 0x7f4fd015c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fd537f640 1 -- 172.21.15.60:0/2762912521 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd01605d0 con 0x7f4fd00ac6e0 2024-05-09T05:00:34.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fd537f640 1 --2- 172.21.15.60:0/2762912521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fd00ac6e0 0x7f4fd0161670 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4fc400bb20 tx=0x7f4fc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fce7fc640 1 -- 172.21.15.60:0/2762912521 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4043020 con 0x7f4fd00ac6e0 2024-05-09T05:00:34.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fce7fc640 1 -- 172.21.15.60:0/2762912521 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fc403f680 con 0x7f4fd00ac6e0 2024-05-09T05:00:34.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fce7fc640 1 -- 172.21.15.60:0/2762912521 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc400b600 con 0x7f4fd00ac6e0 2024-05-09T05:00:34.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fd7de4640 1 -- 172.21.15.60:0/2762912521 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fd01608b0 con 0x7f4fd00ac6e0 2024-05-09T05:00:34.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.239+0000 7f4fd7de4640 1 -- 172.21.15.60:0/2762912521 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd0160e70 con 0x7f4fd00ac6e0 2024-05-09T05:00:34.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.240+0000 7f4fd7de4640 1 -- 172.21.15.60:0/2762912521 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fd0051040 con 0x7f4fd00ac6e0 2024-05-09T05:00:34.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.241+0000 7f4fce7fc640 1 -- 172.21.15.60:0/2762912521 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4fc4046020 con 0x7f4fd00ac6e0 2024-05-09T05:00:34.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.242+0000 7f4fce7fc640 1 --2- 172.21.15.60:0/2762912521 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4fbc076300 0x7f4fbc0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.242+0000 7f4fce7fc640 1 -- 172.21.15.60:0/2762912521 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4fc40b8610 con 0x7f4fd00ac6e0 2024-05-09T05:00:34.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.242+0000 7f4fd5b80640 1 --2- 172.21.15.60:0/2762912521 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4fbc076300 0x7f4fbc0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.242+0000 7f4fd5b80640 1 --2- 172.21.15.60:0/2762912521 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4fbc076300 0x7f4fbc0787c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4fd015d1f0 tx=0x7f4fc8006040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.246+0000 7f4fce7fc640 1 -- 172.21.15.60:0/2762912521 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4fc404d020 con 0x7f4fd00ac6e0 2024-05-09T05:00:34.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.438+0000 7f4fd7de4640 1 -- 172.21.15.60:0/2762912521 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4fd01620a0 con 0x7f4fd00ac6e0 2024-05-09T05:00:34.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.439+0000 7f4fce7fc640 1 -- 172.21.15.60:0/2762912521 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18490 (secure 0 0 0) 0x7f4fc4081cc0 con 0x7f4fd00ac6e0 2024-05-09T05:00:34.440 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:34.441 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:00:17.248958+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:00:10.358133+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:00:11.157769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T05:00:14.850459+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:00:34.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.442+0000 7f4fd7de4640 1 -- 172.21.15.60:0/2762912521 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4fbc076300 msgr2=0x7f4fbc0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.442+0000 7f4fd7de4640 1 --2- 172.21.15.60:0/2762912521 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4fbc076300 0x7f4fbc0787c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4fd015d1f0 tx=0x7f4fc8006040 comp rx=0 tx=0).stop 2024-05-09T05:00:34.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.442+0000 7f4fd7de4640 1 -- 172.21.15.60:0/2762912521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fd00ac6e0 msgr2=0x7f4fd0161670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.442+0000 7f4fd7de4640 1 --2- 172.21.15.60:0/2762912521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fd00ac6e0 0x7f4fd0161670 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4fc400bb20 tx=0x7f4fc40046c0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.442+0000 7f4fd7de4640 1 -- 172.21.15.60:0/2762912521 shutdown_connections 2024-05-09T05:00:34.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.442+0000 7f4fd7de4640 1 --2- 172.21.15.60:0/2762912521 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4fbc076300 0x7f4fbc0787c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.442+0000 7f4fd7de4640 1 --2- 172.21.15.60:0/2762912521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fd01593d0 0x7f4fd015c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.442+0000 7f4fd7de4640 1 --2- 172.21.15.60:0/2762912521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fd00ac6e0 0x7f4fd0161670 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.442+0000 7f4fd7de4640 1 --2- 172.21.15.60:0/2762912521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4fb8004800 0x7f4fd01f5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.442+0000 7f4fd7de4640 1 -- 172.21.15.60:0/2762912521 >> 172.21.15.60:0/2762912521 conn(0x7f4fd00d1470 msgr2=0x7f4fd015f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:34.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.442+0000 7f4fd7de4640 1 -- 172.21.15.60:0/2762912521 shutdown_connections 2024-05-09T05:00:34.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.442+0000 7f4fd7de4640 1 -- 172.21.15.60:0/2762912521 wait complete. 2024-05-09T05:00:34.454 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-05-09T05:00:34.454 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-05-09T05:00:34.455 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-05-09T05:00:34.455 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-05-09T05:00:34.456 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-05-09T05:00:34.456 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-05-09T05:00:34.456 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-05-09T05:00:34.457 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-05-09T05:00:34.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.639+0000 7fb0fadf0640 1 Processor -- start 2024-05-09T05:00:34.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.638+0000 7f4be49f7640 1 Processor -- start 2024-05-09T05:00:34.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.638+0000 7f4be49f7640 1 -- start start 2024-05-09T05:00:34.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.640+0000 7f4be49f7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4bd8101010 0x7f4bd8037c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.640+0000 7f4bdf7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4bd8101010 0x7f4bd8037c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.640+0000 7f4bdf7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4bd8101010 0x7f4bd8037c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34824/0 (socket says 172.21.15.60:34824) 2024-05-09T05:00:34.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.640+0000 7f4bdf7fe640 1 -- 172.21.15.60:0/3686549604 learned_addr learned my addr 172.21.15.60:0/3686549604 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:34.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7f4be49f7640 1 -- 172.21.15.60:0/3686549604 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bd8052cf0 con 0x7f4bd8033920 2024-05-09T05:00:34.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7f4be49f7640 1 -- 172.21.15.60:0/3686549604 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bd80341c0 con 0x7f4bd8054430 2024-05-09T05:00:34.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7f4be49f7640 1 -- 172.21.15.60:0/3686549604 --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f4bd8052f60 con 0x7f4bd8101010 2024-05-09T05:00:34.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.639+0000 7fb0fadf0640 1 -- start start 2024-05-09T05:00:34.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7fb0fadf0640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb0f4071eb0 0x7fb0f4098c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7fb0fadf0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb0f4097e10 0x7fb0f4099150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7fb0fadf0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0f4072f80 con 0x7fb0f4096d50 2024-05-09T05:00:34.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7fb0fadf0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fb0f4098860 con 0x7fb0f4097e10 2024-05-09T05:00:34.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7fb0fadf0640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fb0f4073260 con 0x7fb0f4071eb0 2024-05-09T05:00:34.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7fb0f8b8c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb0f4096d50 0x7fb0f40722b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38452/0 (socket says 172.21.15.60:38452) 2024-05-09T05:00:34.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7fb0f8b8c640 1 -- 172.21.15.60:0/2547121963 learned_addr learned my addr 172.21.15.60:0/2547121963 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:34.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7f4bde7fc640 1 -- 172.21.15.60:0/3686549604 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3668883329 0 0) 0x7f4bd8052cf0 con 0x7f4bd8033920 2024-05-09T05:00:34.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.666+0000 7f4bde7fc640 1 -- 172.21.15.60:0/3686549604 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4bd0003500 con 0x7f4bd8033920 2024-05-09T05:00:34.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7f4bde7fc640 1 -- 172.21.15.60:0/3686549604 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1618849356 0 0) 0x7f4be0071670 con 0x7f4bd8054430 2024-05-09T05:00:34.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7f4bde7fc640 1 -- 172.21.15.60:0/3686549604 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4bd8052cf0 con 0x7f4bd8054430 2024-05-09T05:00:34.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.638+0000 7fb50617f640 1 Processor -- start 2024-05-09T05:00:34.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.638+0000 7fb50617f640 1 -- start start 2024-05-09T05:00:34.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.639+0000 7fb50617f640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb500096d50 0x7fb5000722b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.639+0000 7fb50617f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb500071eb0 0x7fb5000727f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.640+0000 7fb50617f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb500072f80 con 0x7fb500097e10 2024-05-09T05:00:34.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.640+0000 7fb50617f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fb500098860 con 0x7fb500071eb0 2024-05-09T05:00:34.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.640+0000 7fb50617f640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fb500073260 con 0x7fb500096d50 2024-05-09T05:00:34.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.640+0000 7fb4feffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb500071eb0 0x7fb5000727f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.640+0000 7fb4feffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb500071eb0 0x7fb5000727f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34832/0 (socket says 172.21.15.60:34832) 2024-05-09T05:00:34.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7fb0f938d640 1 --2- 172.21.15.60:0/2547121963 >> v2:172.21.15.60:3300/0 conn(0x7fb0f4097e10 0x7fb0f4099150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7fb0ebfff640 1 --2- 172.21.15.60:0/2547121963 >> v2:172.21.15.114:3300/0 conn(0x7fb0f4071eb0 0x7fb0f4098c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7fb0eb7fe640 1 -- 172.21.15.60:0/2547121963 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3197703093 0 0) 0x7fb0f4072f80 con 0x7fb0f4096d50 2024-05-09T05:00:34.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7fb0eb7fe640 1 -- 172.21.15.60:0/2547121963 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0dc003420 con 0x7fb0f4096d50 2024-05-09T05:00:34.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.640+0000 7fb4feffd640 1 -- 172.21.15.60:0/1245399026 learned_addr learned my addr 172.21.15.60:0/1245399026 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:34.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.640+0000 7fb4ff7fe640 1 --2- 172.21.15.60:0/1245399026 >> v2:172.21.15.114:3300/0 conn(0x7fb500096d50 0x7fb5000722b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.640+0000 7fb4fe7fc640 1 -- 172.21.15.60:0/1245399026 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 525898244 0 0) 0x7fb500072f80 con 0x7fb500097e10 2024-05-09T05:00:34.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7fb4fe7fc640 1 -- 172.21.15.60:0/1245399026 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4e8007930 con 0x7fb500097e10 2024-05-09T05:00:34.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.639+0000 7f618e354640 1 Processor -- start 2024-05-09T05:00:34.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.639+0000 7f618e354640 1 -- start start 2024-05-09T05:00:34.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.640+0000 7f618e354640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6180101010 0x7f6180037c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7f618e354640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6180054430 0x7f618004f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7f618cb51640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6180033920 0x7f6180053d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38440/0 (socket says 172.21.15.60:38440) 2024-05-09T05:00:34.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7f618cb51640 1 -- 172.21.15.60:0/1174402332 learned_addr learned my addr 172.21.15.60:0/1174402332 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:34.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7f618e354640 1 -- 172.21.15.60:0/1174402332 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6180052cf0 con 0x7f6180033920 2024-05-09T05:00:34.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7f618d352640 1 --2- 172.21.15.60:0/1174402332 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6180101010 0x7f6180037c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7f618e354640 1 -- 172.21.15.60:0/1174402332 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f61800341c0 con 0x7f6180054430 2024-05-09T05:00:34.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7f618e354640 1 -- 172.21.15.60:0/1174402332 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f61800548c0 con 0x7f6180101010 2024-05-09T05:00:34.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7f618db53640 1 --2- 172.21.15.60:0/1174402332 >> v2:172.21.15.114:3300/0 conn(0x7f6180054430 0x7f618004f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7ff14e493640 1 Processor -- start 2024-05-09T05:00:34.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7ff14e493640 1 -- start start 2024-05-09T05:00:34.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7ff14e493640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff148071eb0 0x7ff148098c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7ff14e493640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff148097e10 0x7ff148099150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7ff14e493640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff148072f80 con 0x7ff148096d50 2024-05-09T05:00:34.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7f617ffff640 1 -- 172.21.15.60:0/1174402332 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1485925717 0 0) 0x7f6180052cf0 con 0x7f6180033920 2024-05-09T05:00:34.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7ff14e493640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ff148098860 con 0x7ff148097e10 2024-05-09T05:00:34.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7ff14e493640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff148073260 con 0x7ff148071eb0 2024-05-09T05:00:34.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7ff147fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff148096d50 0x7ff1480722b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.60:45246/0 (socket says 172.21.15.60:45246) 2024-05-09T05:00:34.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7ff14ca30640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff148097e10 0x7ff148099150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7ff147fff640 1 -- 172.21.15.60:0/516581829 learned_addr learned my addr 172.21.15.60:0/516581829 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:34.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7ff146ffd640 1 -- 172.21.15.60:0/516581829 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4014385530 0 0) 0x7ff148072f80 con 0x7ff148096d50 2024-05-09T05:00:34.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.640+0000 7f8188ce4640 1 Processor -- start 2024-05-09T05:00:34.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.640+0000 7f8188ce4640 1 -- start start 2024-05-09T05:00:34.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7f8188ce4640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f817c101010 0x7f817c037c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7f8188ce4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f817c033920 0x7f817c053d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7f8188ce4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f817c052cf0 con 0x7f817c054260 2024-05-09T05:00:34.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7f8188ce4640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f817c0341c0 con 0x7f817c101010 2024-05-09T05:00:34.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7f8188ce4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f817c052f60 con 0x7f817c033920 2024-05-09T05:00:34.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7f81837fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f817c101010 0x7f817c037c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7f8183fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f817c054260 0x7f817c005e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.60:45242/0 (socket says 172.21.15.60:45242) 2024-05-09T05:00:34.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7f8183fff640 1 -- 172.21.15.60:0/3908824373 learned_addr learned my addr 172.21.15.60:0/3908824373 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:34.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7f666d4b4640 1 Processor -- start 2024-05-09T05:00:34.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.641+0000 7f666d4b4640 1 -- start start 2024-05-09T05:00:34.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7f666d4b4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6668097c40 0x7f66680b0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.643+0000 7f6666ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6668097c40 0x7f66680b0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.643+0000 7f66677fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6668098ac0 0x7f6668073a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38456/0 (socket says 172.21.15.60:38456) 2024-05-09T05:00:34.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7ff1477fe640 1 --2- 172.21.15.60:0/516581829 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff148071eb0 0x7ff148098c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7f8182ffd640 1 --2- 172.21.15.60:0/3908824373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f817c033920 0x7f817c053d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.642+0000 7f81827fc640 1 -- 172.21.15.60:0/3908824373 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2943277187 0 0) 0x7f817c052cf0 con 0x7f817c054260 2024-05-09T05:00:34.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7f81827fc640 1 -- 172.21.15.60:0/3908824373 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8174003520 con 0x7f817c054260 2024-05-09T05:00:34.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.643+0000 7f66677fe640 1 -- 172.21.15.60:0/982128318 learned_addr learned my addr 172.21.15.60:0/982128318 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:34.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.643+0000 7f666d4b4640 1 -- 172.21.15.60:0/982128318 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6668099000 con 0x7f6668098ac0 2024-05-09T05:00:34.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.643+0000 7f666d4b4640 1 -- 172.21.15.60:0/982128318 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66680992e0 con 0x7f666816c5a0 2024-05-09T05:00:34.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.643+0000 7f666d4b4640 1 -- 172.21.15.60:0/982128318 --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f66680735e0 con 0x7f6668097c40 2024-05-09T05:00:34.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.643+0000 7ff525431640 1 Processor -- start 2024-05-09T05:00:34.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.643+0000 7ff525431640 1 -- start start 2024-05-09T05:00:34.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.644+0000 7ff525431640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff5200962f0 0x7ff52009ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.644+0000 7ff525431640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff52009b160 0x7ff5200966d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.644+0000 7ff525431640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff520096c10 0x7ff520097060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.644+0000 7ff525431640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7ff5200d0d00 con 0x7ff52009b160 2024-05-09T05:00:34.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.644+0000 7ff525431640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ff520074270 con 0x7ff5200962f0 2024-05-09T05:00:34.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.644+0000 7ff525431640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff520074550 con 0x7ff520096c10 2024-05-09T05:00:34.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.644+0000 7ff517fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff52009b160 0x7ff5200966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.644+0000 7ff517fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff52009b160 0x7ff5200966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34862/0 (socket says 172.21.15.60:34862) 2024-05-09T05:00:34.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.644+0000 7ff51ffff640 1 --2- 172.21.15.60:0/71837445 >> v2:172.21.15.114:3300/0 conn(0x7ff5200962f0 0x7ff52009ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.644+0000 7ff524c30640 1 --2- 172.21.15.60:0/71837445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff520096c10 0x7ff520097060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.644+0000 7ff517fff640 1 -- 172.21.15.60:0/71837445 learned_addr learned my addr 172.21.15.60:0/71837445 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:34.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7fb4ff7fe640 1 -- 172.21.15.60:0/1245399026 >> v2:172.21.15.60:3300/0 conn(0x7fb500071eb0 msgr2=0x7fb5000727f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:34.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7fb4ff7fe640 1 --2- 172.21.15.60:0/1245399026 >> v2:172.21.15.60:3300/0 conn(0x7fb500071eb0 0x7fb5000727f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7fb4ff7fe640 1 -- 172.21.15.60:0/1245399026 >> v1:172.21.15.114:6789/0 conn(0x7fb500097e10 legacy=0x7fb50006e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7fb4ff7fe640 1 -- 172.21.15.60:0/1245399026 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb500070630 con 0x7fb500096d50 2024-05-09T05:00:34.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.643+0000 7f6665ffb640 1 -- 172.21.15.60:0/982128318 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 236056273 0 0) 0x7f6668099000 con 0x7f6668098ac0 2024-05-09T05:00:34.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7f617ffff640 1 -- 172.21.15.60:0/1174402332 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6174003500 con 0x7f6180033920 2024-05-09T05:00:34.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7f6665ffb640 1 -- 172.21.15.60:0/982128318 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6650003520 con 0x7f6668098ac0 2024-05-09T05:00:34.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7f6665ffb640 1 -- 172.21.15.60:0/982128318 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2403405869 0 0) 0x7f66680992e0 con 0x7f666816c5a0 2024-05-09T05:00:34.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7f6665ffb640 1 -- 172.21.15.60:0/982128318 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6668099000 con 0x7f666816c5a0 2024-05-09T05:00:34.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7f4bdf7fe640 1 -- 172.21.15.60:0/3686549604 >> v1:172.21.15.114:6789/0 conn(0x7f4bd8054430 legacy=0x7f4bd804f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7f4bdf7fe640 1 -- 172.21.15.60:0/3686549604 >> v1:172.21.15.60:6789/0 conn(0x7f4bd8033920 legacy=0x7f4bd8053d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7f4bdf7fe640 1 -- 172.21.15.60:0/3686549604 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd8053240 con 0x7f4bd8101010 2024-05-09T05:00:34.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.668+0000 7ff146ffd640 1 -- 172.21.15.60:0/516581829 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff130003970 con 0x7ff148096d50 2024-05-09T05:00:34.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.668+0000 7ff51ffff640 1 -- 172.21.15.60:0/71837445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff520096c10 msgr2=0x7ff520097060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:34.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.668+0000 7ff51ffff640 1 --2- 172.21.15.60:0/71837445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff520096c10 0x7ff520097060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.668+0000 7ff51ffff640 1 -- 172.21.15.60:0/71837445 >> v2:172.21.15.60:3300/0 conn(0x7ff52009b160 msgr2=0x7ff5200966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:34.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.668+0000 7ff51ffff640 1 --2- 172.21.15.60:0/71837445 >> v2:172.21.15.60:3300/0 conn(0x7ff52009b160 0x7ff5200966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.668+0000 7ff14ca30640 1 -- 172.21.15.60:0/516581829 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff148071eb0 msgr2=0x7ff148098c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:34.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.668+0000 7ff14ca30640 1 --2- 172.21.15.60:0/516581829 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff148071eb0 0x7ff148098c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.668+0000 7f8182ffd640 1 -- 172.21.15.60:0/3908824373 >> v2:172.21.15.114:3300/0 conn(0x7f817c101010 msgr2=0x7f817c037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:34.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.668+0000 7f8182ffd640 1 --2- 172.21.15.60:0/3908824373 >> v2:172.21.15.114:3300/0 conn(0x7f817c101010 0x7f817c037c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.668+0000 7f8182ffd640 1 -- 172.21.15.60:0/3908824373 >> v1:172.21.15.114:6789/0 conn(0x7f817c054260 legacy=0x7f817c005e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.668+0000 7f8182ffd640 1 -- 172.21.15.60:0/3908824373 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817c053240 con 0x7f817c033920 2024-05-09T05:00:34.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7fb0f938d640 1 -- 172.21.15.60:0/2547121963 >> v2:172.21.15.114:3300/0 conn(0x7fb0f4071eb0 msgr2=0x7fb0f4098c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:34.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7fb0f938d640 1 --2- 172.21.15.60:0/2547121963 >> v2:172.21.15.114:3300/0 conn(0x7fb0f4071eb0 0x7fb0f4098c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7fb0f938d640 1 -- 172.21.15.60:0/2547121963 >> v1:172.21.15.60:6789/0 conn(0x7fb0f4096d50 legacy=0x7fb0f40722b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7fb0f938d640 1 -- 172.21.15.60:0/2547121963 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0f4070630 con 0x7fb0f4097e10 2024-05-09T05:00:34.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.668+0000 7ff14ca30640 1 -- 172.21.15.60:0/516581829 >> v1:172.21.15.114:6789/0 conn(0x7ff148096d50 legacy=0x7ff1480722b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7f618d352640 1 -- 172.21.15.60:0/1174402332 >> v2:172.21.15.114:3300/0 conn(0x7f6180054430 msgr2=0x7f618004f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:34.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.667+0000 7f618d352640 1 --2- 172.21.15.60:0/1174402332 >> v2:172.21.15.114:3300/0 conn(0x7f6180054430 0x7f618004f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.668+0000 7f618d352640 1 -- 172.21.15.60:0/1174402332 >> v1:172.21.15.60:6789/0 conn(0x7f6180033920 legacy=0x7f6180053d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.668+0000 7f618d352640 1 -- 172.21.15.60:0/1174402332 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6180052f60 con 0x7f6180101010 2024-05-09T05:00:34.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.668+0000 7ff14ca30640 1 -- 172.21.15.60:0/516581829 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff148070630 con 0x7ff148097e10 2024-05-09T05:00:34.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.669+0000 7fb0f938d640 1 --2- 172.21.15.60:0/2547121963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0f4097e10 0x7fb0f4099150 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb0f40b4830 tx=0x7fb0ec02a690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ee20b28f922cea server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.669+0000 7f4bdf7fe640 1 --2- 172.21.15.60:0/3686549604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bd8101010 0x7f4bd8037c40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009870 tx=0x7f4bd402a3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d0ce9b081513df8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.669+0000 7f618d352640 1 --2- 172.21.15.60:0/1174402332 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6180101010 0x7f6180037c40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f61780071d0 tx=0x7f617802b230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a19be2dd7c299f7d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.668+0000 7ff51ffff640 1 -- 172.21.15.60:0/71837445 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff520074830 con 0x7ff5200962f0 2024-05-09T05:00:34.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.668+0000 7f6666ffd640 1 -- 172.21.15.60:0/982128318 >> v1:172.21.15.114:6789/0 conn(0x7f666816c5a0 legacy=0x7f66680983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.668+0000 7f6666ffd640 1 -- 172.21.15.60:0/982128318 >> v1:172.21.15.60:6789/0 conn(0x7f6668098ac0 legacy=0x7f6668073a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.668+0000 7f6666ffd640 1 -- 172.21.15.60:0/982128318 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66680995c0 con 0x7f6668097c40 2024-05-09T05:00:34.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.669+0000 7f4bde7fc640 1 -- 172.21.15.60:0/3686549604 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4037070 con 0x7f4bd8101010 2024-05-09T05:00:34.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.669+0000 7f617ffff640 1 -- 172.21.15.60:0/1174402332 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61780089e0 con 0x7f6180101010 2024-05-09T05:00:34.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.669+0000 7f617ffff640 1 -- 172.21.15.60:0/1174402332 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6178008b80 con 0x7f6180101010 2024-05-09T05:00:34.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.669+0000 7f4bde7fc640 1 -- 172.21.15.60:0/3686549604 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bd4032640 con 0x7f4bd8101010 2024-05-09T05:00:34.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.669+0000 7f617ffff640 1 -- 172.21.15.60:0/1174402332 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617800a0d0 con 0x7f6180101010 2024-05-09T05:00:34.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.669+0000 7f8182ffd640 1 --2- 172.21.15.60:0/3908824373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f817c033920 0x7f817c053d20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f818406ab40 tx=0x7f81840d0fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e7fbcc3ae874ac8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.669+0000 7f4bde7fc640 1 -- 172.21.15.60:0/3686549604 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd40329f0 con 0x7f4bd8101010 2024-05-09T05:00:34.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.669+0000 7f6666ffd640 1 --2- 172.21.15.60:0/982128318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6668097c40 0x7f66680b0fc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f66580097d0 tx=0x7f6658029f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f34ed7432cf54e13 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.669+0000 7ff51ffff640 1 --2- 172.21.15.60:0/71837445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5200962f0 0x7ff52009ac20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff52006e1e0 tx=0x7ff51000a370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce1acfe04c235c4c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.669+0000 7fb0eb7fe640 1 -- 172.21.15.60:0/2547121963 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec036070 con 0x7fb0f4097e10 2024-05-09T05:00:34.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.669+0000 7fb0eb7fe640 1 -- 172.21.15.60:0/2547121963 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0ec02f710 con 0x7fb0f4097e10 2024-05-09T05:00:34.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.669+0000 7fb0eb7fe640 1 -- 172.21.15.60:0/2547121963 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec02fb20 con 0x7fb0f4097e10 2024-05-09T05:00:34.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.669+0000 7f81827fc640 1 -- 172.21.15.60:0/3908824373 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818407e8e0 con 0x7f817c033920 2024-05-09T05:00:34.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.669+0000 7fb4ff7fe640 1 --2- 172.21.15.60:0/1245399026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb500096d50 0x7fb5000722b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb5000690e0 tx=0x7fb4f002a850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7605955df90e087a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.669+0000 7f81827fc640 1 -- 172.21.15.60:0/3908824373 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f818407ea80 con 0x7f817c033920 2024-05-09T05:00:34.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7ff14ca30640 1 --2- 172.21.15.60:0/516581829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff148097e10 0x7ff148099150 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff140009d90 tx=0x7ff14002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33d280f932d9ad14 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7fb4fe7fc640 1 -- 172.21.15.60:0/1245399026 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0037070 con 0x7fb500096d50 2024-05-09T05:00:34.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7fb4fe7fc640 1 -- 172.21.15.60:0/1245399026 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4f002e050 con 0x7fb500096d50 2024-05-09T05:00:34.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7fb4fe7fc640 1 -- 172.21.15.60:0/1245399026 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0004490 con 0x7fb500096d50 2024-05-09T05:00:34.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.669+0000 7f81827fc640 1 -- 172.21.15.60:0/3908824373 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818407ee30 con 0x7f817c033920 2024-05-09T05:00:34.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7f6665ffb640 1 -- 172.21.15.60:0/982128318 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665802aab0 con 0x7f6668097c40 2024-05-09T05:00:34.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7f6665ffb640 1 -- 172.21.15.60:0/982128318 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f665802ac50 con 0x7f6668097c40 2024-05-09T05:00:34.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7f6665ffb640 1 -- 172.21.15.60:0/982128318 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66580092e0 con 0x7f6668097c40 2024-05-09T05:00:34.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7f618e354640 1 -- 172.21.15.60:0/1174402332 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6180101010 msgr2=0x7f6180037c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7f618e354640 1 --2- 172.21.15.60:0/1174402332 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6180101010 0x7f6180037c40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f61780071d0 tx=0x7f617802b230 comp rx=0 tx=0).stop 2024-05-09T05:00:34.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7ff146ffd640 1 -- 172.21.15.60:0/516581829 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff140007740 con 0x7ff148097e10 2024-05-09T05:00:34.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7ff146ffd640 1 -- 172.21.15.60:0/516581829 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1400078e0 con 0x7ff148097e10 2024-05-09T05:00:34.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7ff146ffd640 1 -- 172.21.15.60:0/516581829 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff140007740 con 0x7ff148097e10 2024-05-09T05:00:34.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7f618e354640 1 -- 172.21.15.60:0/1174402332 shutdown_connections 2024-05-09T05:00:34.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7f618e354640 1 --2- 172.21.15.60:0/1174402332 >> v2:172.21.15.114:3300/0 conn(0x7f6180054430 0x7f618004f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7fb0fadf0640 1 -- 172.21.15.60:0/2547121963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0f4097e10 msgr2=0x7fb0f4099150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7fb0fadf0640 1 --2- 172.21.15.60:0/2547121963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0f4097e10 0x7fb0f4099150 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb0f40b4830 tx=0x7fb0ec02a690 comp rx=0 tx=0).stop 2024-05-09T05:00:34.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7f8188ce4640 1 -- 172.21.15.60:0/3908824373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f817c033920 msgr2=0x7f817c053d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7f8188ce4640 1 --2- 172.21.15.60:0/3908824373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f817c033920 0x7f817c053d20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f818406ab40 tx=0x7f81840d0fc0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7f4be49f7640 1 -- 172.21.15.60:0/3686549604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bd8101010 msgr2=0x7f4bd8037c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7f4be49f7640 1 --2- 172.21.15.60:0/3686549604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bd8101010 0x7f4bd8037c40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009870 tx=0x7f4bd402a3c0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7f4be49f7640 1 -- 172.21.15.60:0/3686549604 shutdown_connections 2024-05-09T05:00:34.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7f4be49f7640 1 --2- 172.21.15.60:0/3686549604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bd8101010 0x7f4bd8037c40 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009870 tx=0x7f4bd402a3c0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7f4be49f7640 1 -- 172.21.15.60:0/3686549604 >> 172.21.15.60:0/3686549604 conn(0x7f4bd8009fb0 msgr2=0x7f4bd800a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:34.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7ff51f7fe640 1 -- 172.21.15.60:0/71837445 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff510030070 con 0x7ff5200962f0 2024-05-09T05:00:34.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7f618e354640 1 --2- 172.21.15.60:0/1174402332 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6180101010 0x7f6180037c40 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.670+0000 7f618e354640 1 -- 172.21.15.60:0/1174402332 >> 172.21.15.60:0/1174402332 conn(0x7f6180009fb0 msgr2=0x7f618000a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:34.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7f618e354640 1 -- 172.21.15.60:0/1174402332 shutdown_connections 2024-05-09T05:00:34.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7f618e354640 1 -- 172.21.15.60:0/1174402332 wait complete. 2024-05-09T05:00:34.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7ff51f7fe640 1 -- 172.21.15.60:0/71837445 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff510034400 con 0x7ff5200962f0 2024-05-09T05:00:34.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7f8188ce4640 1 -- 172.21.15.60:0/3908824373 shutdown_connections 2024-05-09T05:00:34.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7f8188ce4640 1 --2- 172.21.15.60:0/3908824373 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f817c033920 0x7f817c053d20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7f8188ce4640 1 --2- 172.21.15.60:0/3908824373 >> v2:172.21.15.114:3300/0 conn(0x7f817c101010 0x7f817c037c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7f8188ce4640 1 -- 172.21.15.60:0/3908824373 >> 172.21.15.60:0/3908824373 conn(0x7f817c009fb0 msgr2=0x7f817c00a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:34.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7f666d4b4640 1 -- 172.21.15.60:0/982128318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6668097c40 msgr2=0x7f66680b0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7f4be49f7640 1 -- 172.21.15.60:0/3686549604 shutdown_connections 2024-05-09T05:00:34.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7f666d4b4640 1 --2- 172.21.15.60:0/982128318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6668097c40 0x7f66680b0fc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f66580097d0 tx=0x7f6658029f70 comp rx=0 tx=0).stop 2024-05-09T05:00:34.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7f4be49f7640 1 -- 172.21.15.60:0/3686549604 wait complete. 2024-05-09T05:00:34.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7ff51f7fe640 1 -- 172.21.15.60:0/71837445 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff510030070 con 0x7ff5200962f0 2024-05-09T05:00:34.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7f666d4b4640 1 -- 172.21.15.60:0/982128318 shutdown_connections 2024-05-09T05:00:34.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7f666d4b4640 1 --2- 172.21.15.60:0/982128318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6668097c40 0x7f66680b0fc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7f666d4b4640 1 -- 172.21.15.60:0/982128318 >> 172.21.15.60:0/982128318 conn(0x7f66680afbd0 msgr2=0x7f66680705e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:34.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7f666d4b4640 1 -- 172.21.15.60:0/982128318 shutdown_connections 2024-05-09T05:00:34.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7f666d4b4640 1 -- 172.21.15.60:0/982128318 wait complete. 2024-05-09T05:00:34.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7f8188ce4640 1 -- 172.21.15.60:0/3908824373 shutdown_connections 2024-05-09T05:00:34.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7f8188ce4640 1 -- 172.21.15.60:0/3908824373 wait complete. 2024-05-09T05:00:34.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7fb0fadf0640 1 -- 172.21.15.60:0/2547121963 shutdown_connections 2024-05-09T05:00:34.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7fb0fadf0640 1 --2- 172.21.15.60:0/2547121963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0f4097e10 0x7fb0f4099150 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7fb0fadf0640 1 --2- 172.21.15.60:0/2547121963 >> v2:172.21.15.114:3300/0 conn(0x7fb0f4071eb0 0x7fb0f4098c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7fb0fadf0640 1 -- 172.21.15.60:0/2547121963 >> 172.21.15.60:0/2547121963 conn(0x7fb0f40972f0 msgr2=0x7fb0f4097700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:34.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.672+0000 7ff525431640 1 -- 172.21.15.60:0/71837445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5200962f0 msgr2=0x7ff52009ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7ff14e493640 1 -- 172.21.15.60:0/516581829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff148097e10 msgr2=0x7ff148099150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7ff14e493640 1 --2- 172.21.15.60:0/516581829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff148097e10 0x7ff148099150 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff140009d90 tx=0x7ff14002a660 comp rx=0 tx=0).stop 2024-05-09T05:00:34.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7ff14e493640 1 -- 172.21.15.60:0/516581829 shutdown_connections 2024-05-09T05:00:34.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7ff14e493640 1 --2- 172.21.15.60:0/516581829 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff148097e10 0x7ff148099150 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7ff14e493640 1 --2- 172.21.15.60:0/516581829 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff148071eb0 0x7ff148098c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7ff14e493640 1 -- 172.21.15.60:0/516581829 >> 172.21.15.60:0/516581829 conn(0x7ff1480972f0 msgr2=0x7ff148097700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:34.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7ff14e493640 1 -- 172.21.15.60:0/516581829 shutdown_connections 2024-05-09T05:00:34.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7ff14e493640 1 -- 172.21.15.60:0/516581829 wait complete. 2024-05-09T05:00:34.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.672+0000 7fb0fadf0640 1 -- 172.21.15.60:0/2547121963 shutdown_connections 2024-05-09T05:00:34.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.672+0000 7fb0fadf0640 1 -- 172.21.15.60:0/2547121963 wait complete. 2024-05-09T05:00:34.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.672+0000 7ff525431640 1 --2- 172.21.15.60:0/71837445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5200962f0 0x7ff52009ac20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff52006e1e0 tx=0x7ff51000a370 comp rx=0 tx=0).stop 2024-05-09T05:00:34.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.672+0000 7ff525431640 1 -- 172.21.15.60:0/71837445 shutdown_connections 2024-05-09T05:00:34.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.672+0000 7ff525431640 1 --2- 172.21.15.60:0/71837445 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff520096c10 0x7ff520097060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.672+0000 7ff525431640 1 --2- 172.21.15.60:0/71837445 >> v2:172.21.15.60:3300/0 conn(0x7ff52009b160 0x7ff5200966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.672+0000 7ff525431640 1 --2- 172.21.15.60:0/71837445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5200962f0 0x7ff52009ac20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.673+0000 7f618e354640 1 Processor -- start 2024-05-09T05:00:34.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.672+0000 7ff525431640 1 -- 172.21.15.60:0/71837445 >> 172.21.15.60:0/71837445 conn(0x7ff5200b0940 msgr2=0x7ff5200b0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:34.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.673+0000 7f4be49f7640 1 Processor -- start 2024-05-09T05:00:34.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.673+0000 7ff14e493640 1 Processor -- start 2024-05-09T05:00:34.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.673+0000 7f8188ce4640 1 Processor -- start 2024-05-09T05:00:34.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.673+0000 7ff525431640 1 -- 172.21.15.60:0/71837445 shutdown_connections 2024-05-09T05:00:34.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.673+0000 7ff525431640 1 -- 172.21.15.60:0/71837445 wait complete. 2024-05-09T05:00:34.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.673+0000 7f666d4b4640 1 Processor -- start 2024-05-09T05:00:34.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.674+0000 7ff14e493640 1 -- start start 2024-05-09T05:00:34.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.674+0000 7f8188ce4640 1 -- start start 2024-05-09T05:00:34.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.674+0000 7fb0fadf0640 1 Processor -- start 2024-05-09T05:00:34.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.674+0000 7f4be49f7640 1 -- start start 2024-05-09T05:00:34.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.674+0000 7fb0fadf0640 1 -- start start 2024-05-09T05:00:34.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.673+0000 7f618e354640 1 -- start start 2024-05-09T05:00:34.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.674+0000 7f618e354640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6180033920 0x7f6180103120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.674+0000 7f618e354640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6180054430 0x7f61800f3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7fb50617f640 1 -- 172.21.15.60:0/1245399026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb500096d50 msgr2=0x7fb5000722b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7fb50617f640 1 --2- 172.21.15.60:0/1245399026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb500096d50 0x7fb5000722b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb5000690e0 tx=0x7fb4f002a850 comp rx=0 tx=0).stop 2024-05-09T05:00:34.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7fb50617f640 1 -- 172.21.15.60:0/1245399026 shutdown_connections 2024-05-09T05:00:34.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7fb50617f640 1 --2- 172.21.15.60:0/1245399026 >> v2:172.21.15.60:3300/0 conn(0x7fb500071eb0 0x7fb5000727f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7fb50617f640 1 --2- 172.21.15.60:0/1245399026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb500096d50 0x7fb5000722b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7fb50617f640 1 -- 172.21.15.60:0/1245399026 >> 172.21.15.60:0/1245399026 conn(0x7fb5000972f0 msgr2=0x7fb500097700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:34.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7fb50617f640 1 -- 172.21.15.60:0/1245399026 shutdown_connections 2024-05-09T05:00:34.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.671+0000 7fb50617f640 1 -- 172.21.15.60:0/1245399026 wait complete. 2024-05-09T05:00:34.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.673+0000 7fb50617f640 1 Processor -- start 2024-05-09T05:00:34.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.673+0000 7fb50617f640 1 -- start start 2024-05-09T05:00:34.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.674+0000 7fb50617f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb500071eb0 0x7fb5001f4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.674+0000 7fb50617f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb500096d50 0x7fb5001f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.674+0000 7fb4ff7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb500071eb0 0x7fb5001f4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.674+0000 7fb4ff7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb500071eb0 0x7fb5001f4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34876/0 (socket says 172.21.15.60:34876) 2024-05-09T05:00:34.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.674+0000 7fb50617f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb500097e10 0x7fb5001f7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.674+0000 7fb50617f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb500099c20 con 0x7fb500071eb0 2024-05-09T05:00:34.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.674+0000 7fb50617f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb500072d30 con 0x7fb500097e10 2024-05-09T05:00:34.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.674+0000 7fb50617f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb500099da0 con 0x7fb500096d50 2024-05-09T05:00:34.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.674+0000 7fb4ff7fe640 1 -- 172.21.15.60:0/1698444652 learned_addr learned my addr 172.21.15.60:0/1698444652 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:34.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7fb4fffff640 1 --2- 172.21.15.60:0/1698444652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb500097e10 0x7fb5001f7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7ff14e493640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff148071eb0 0x7ff148093950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.674+0000 7f666d4b4640 1 -- start start 2024-05-09T05:00:34.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7ff14e493640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff148096d50 0x7ff14816a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7f8188ce4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f817c033920 0x7f817c196f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7f8188ce4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f817c054260 0x7f817c0f8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7f666d4b4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6668097c40 0x7f66680ac650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7f666d4b4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6668098ac0 0x7f666820a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7f666d4b4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f666816c5a0 0x7f666820add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7f666d4b4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f66680b1680 con 0x7f666816c5a0 2024-05-09T05:00:34.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7fb4feffd640 1 --2- 172.21.15.60:0/1698444652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb500096d50 0x7fb5001f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7fb4fffff640 1 -- 172.21.15.60:0/1698444652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb500096d50 msgr2=0x7fb5001f6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7f666d4b4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f66680b1500 con 0x7f6668097c40 2024-05-09T05:00:34.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7f666d4b4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f66680b1800 con 0x7f6668098ac0 2024-05-09T05:00:34.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7f66677fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f666816c5a0 0x7f666820add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7f66677fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f666816c5a0 0x7f666820add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34890/0 (socket says 172.21.15.60:34890) 2024-05-09T05:00:34.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7f66677fe640 1 -- 172.21.15.60:0/3394702519 learned_addr learned my addr 172.21.15.60:0/3394702519 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:34.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7ff147fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff148071eb0 0x7ff148093950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7ff14e493640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff148097e10 0x7ff14816a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7fb4fffff640 1 --2- 172.21.15.60:0/1698444652 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb500096d50 0x7fb5001f6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7fb4fffff640 1 -- 172.21.15.60:0/1698444652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb500071eb0 msgr2=0x7fb5001f4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7ff147fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff148071eb0 0x7ff148093950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.60:41804/0 (socket says 172.21.15.60:41804) 2024-05-09T05:00:34.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7ff147fff640 1 -- 172.21.15.60:0/1638895201 learned_addr learned my addr 172.21.15.60:0/1638895201 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:34.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7f8188ce4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f817c101010 0x7f817c0fa680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7f4be49f7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bd8033920 0x7f4bd80f2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7ff1477fe640 1 --2- 172.21.15.60:0/1638895201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff148096d50 0x7ff14816a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7ff14ca30640 1 --2- 172.21.15.60:0/1638895201 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff148097e10 0x7ff14816a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f4be49f7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bd8054430 0x7f4bd802dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7ff14e493640 1 -- 172.21.15.60:0/1638895201 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff148072b40 con 0x7ff148096d50 2024-05-09T05:00:34.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f4be49f7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bd802e1c0 0x7f4bd804e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7ff14e493640 1 -- 172.21.15.60:0/1638895201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff1480729c0 con 0x7ff148071eb0 2024-05-09T05:00:34.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7ff14e493640 1 -- 172.21.15.60:0/1638895201 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff148072cc0 con 0x7ff148097e10 2024-05-09T05:00:34.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7f618d352640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6180033920 0x7f6180103120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f66677fe640 1 -- 172.21.15.60:0/3394702519 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6668098ac0 msgr2=0x7f666820a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T05:00:34.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f4be49f7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd8054860 con 0x7f4bd8054430 2024-05-09T05:00:34.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f66677fe640 1 --2- 172.21.15.60:0/3394702519 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6668098ac0 0x7f666820a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f66677fe640 1 -- 172.21.15.60:0/3394702519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6668097c40 msgr2=0x7f66680ac650 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T05:00:34.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f66677fe640 1 --2- 172.21.15.60:0/3394702519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6668097c40 0x7f66680ac650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f66677fe640 1 -- 172.21.15.60:0/3394702519 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66680acf80 con 0x7f666816c5a0 2024-05-09T05:00:34.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f8188ce4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f817c0502a0 con 0x7f817c033920 2024-05-09T05:00:34.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f8188ce4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f817c053800 con 0x7f817c054260 2024-05-09T05:00:34.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f4bdffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bd802e1c0 0x7f4bd804e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f4bdffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bd802e1c0 0x7f4bd804e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.60:41830/0 (socket says 172.21.15.60:41830) 2024-05-09T05:00:34.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f4bdffff640 1 -- 172.21.15.60:0/2161943980 learned_addr learned my addr 172.21.15.60:0/2161943980 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:34.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f8188ce4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f817c038180 con 0x7f817c101010 2024-05-09T05:00:34.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f618e354640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6180101010 0x7f61801a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f4be49f7640 1 -- 172.21.15.60:0/2161943980 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd8053800 con 0x7f4bd802e1c0 2024-05-09T05:00:34.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f4be49f7640 1 -- 172.21.15.60:0/2161943980 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4bd8038180 con 0x7f4bd8033920 2024-05-09T05:00:34.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f4bdeffd640 1 --2- 172.21.15.60:0/2161943980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bd8054430 0x7f4bd802dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f618e354640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6180053800 con 0x7f6180033920 2024-05-09T05:00:34.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.675+0000 7fb4fffff640 1 --2- 172.21.15.60:0/1698444652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb500071eb0 0x7fb5001f4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f8182ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f817c054260 0x7f817c0f8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7ff525431640 1 Processor -- start 2024-05-09T05:00:34.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7ff525431640 1 -- start start 2024-05-09T05:00:34.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f66677fe640 1 --2- 172.21.15.60:0/3394702519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f666816c5a0 0x7f666820add0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f666000c7a0 tx=0x7f666000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7fb0fadf0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f4071eb0 0x7fb0f4091a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7ff147fff640 1 -- 172.21.15.60:0/1638895201 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff148097e10 msgr2=0x7ff14816a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7ff147fff640 1 --2- 172.21.15.60:0/1638895201 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff148097e10 0x7ff14816a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7ff147fff640 1 -- 172.21.15.60:0/1638895201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff148096d50 msgr2=0x7ff14816a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7ff147fff640 1 --2- 172.21.15.60:0/1638895201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff148096d50 0x7ff14816a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7ff147fff640 1 -- 172.21.15.60:0/1638895201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff14816b1a0 con 0x7ff148071eb0 2024-05-09T05:00:34.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f8182ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f817c054260 0x7f817c0f8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.60:41850/0 (socket says 172.21.15.60:41850) 2024-05-09T05:00:34.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f8182ffd640 1 -- 172.21.15.60:0/3741007679 learned_addr learned my addr 172.21.15.60:0/3741007679 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:34.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7fb0fadf0640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f4096d50 0x7fb0f40cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f4bdf7fe640 1 --2- 172.21.15.60:0/2161943980 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bd8033920 0x7f4bd80f2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f81837fe640 1 --2- 172.21.15.60:0/3741007679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f817c033920 0x7f817c196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7fb0fadf0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0f4097e10 0x7fb0f40aadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f664f7fe640 1 -- 172.21.15.60:0/3394702519 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6660018070 con 0x7f666816c5a0 2024-05-09T05:00:34.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7fb0fadf0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f4072b40 con 0x7fb0f4097e10 2024-05-09T05:00:34.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7ff147fff640 1 --2- 172.21.15.60:0/1638895201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff148071eb0 0x7ff148093950 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff13800c500 tx=0x7ff13800c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7fb0fadf0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f40729c0 con 0x7fb0f4071eb0 2024-05-09T05:00:34.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7fb0fadf0640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb0f4072cc0 con 0x7fb0f4096d50 2024-05-09T05:00:34.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f664f7fe640 1 -- 172.21.15.60:0/3394702519 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f666000a640 con 0x7f666816c5a0 2024-05-09T05:00:34.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f666d4b4640 1 -- 172.21.15.60:0/3394702519 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66680ad260 con 0x7f666816c5a0 2024-05-09T05:00:34.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7fb0f8b8c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f4071eb0 0x7fb0f4091a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f618db53640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6180101010 0x7f61801a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7f618cb51640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6180054430 0x7f61800f3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f618db53640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6180101010 0x7f61801a2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.60:41834/0 (socket says 172.21.15.60:41834) 2024-05-09T05:00:34.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f618db53640 1 -- 172.21.15.60:0/1887603497 learned_addr learned my addr 172.21.15.60:0/1887603497 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:34.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f666d4b4640 1 -- 172.21.15.60:0/3394702519 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66680ad820 con 0x7f666816c5a0 2024-05-09T05:00:34.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7fb4fffff640 1 -- 172.21.15.60:0/1698444652 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5001f50c0 con 0x7fb500097e10 2024-05-09T05:00:34.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f4bdffff640 1 -- 172.21.15.60:0/2161943980 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bd8033920 msgr2=0x7f4bd80f2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f4bdffff640 1 --2- 172.21.15.60:0/2161943980 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4bd8033920 0x7f4bd80f2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f4bdffff640 1 -- 172.21.15.60:0/2161943980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bd8054430 msgr2=0x7f4bd802dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.676+0000 7fb4ff7fe640 1 --2- 172.21.15.60:0/1698444652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb500071eb0 0x7fb5001f4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:00:34.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f664f7fe640 1 -- 172.21.15.60:0/3394702519 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666000a9f0 con 0x7f666816c5a0 2024-05-09T05:00:34.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f4bdffff640 1 --2- 172.21.15.60:0/2161943980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bd8054430 0x7f4bd802dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f4bdffff640 1 -- 172.21.15.60:0/2161943980 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd804f550 con 0x7f4bd802e1c0 2024-05-09T05:00:34.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7ff525431640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5200962f0 0x7ff5200a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f618e354640 1 -- 172.21.15.60:0/1887603497 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6180053520 con 0x7f6180101010 2024-05-09T05:00:34.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7ff525431640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff520096c10 0x7ff520201210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f8182ffd640 1 -- 172.21.15.60:0/3741007679 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f817c101010 msgr2=0x7f817c0fa680 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T05:00:34.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f8182ffd640 1 --2- 172.21.15.60:0/3741007679 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f817c101010 0x7f817c0fa680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f4bdeffd640 1 --2- 172.21.15.60:0/2161943980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4bd8054430 0x7f4bd802dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:34.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f8182ffd640 1 -- 172.21.15.60:0/3741007679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f817c033920 msgr2=0x7f817c196f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f618e354640 1 -- 172.21.15.60:0/1887603497 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6180038180 con 0x7f6180054430 2024-05-09T05:00:34.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7f8182ffd640 1 --2- 172.21.15.60:0/3741007679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f817c033920 0x7f817c196f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7ff51ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5200962f0 0x7ff5200a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7ff51ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5200962f0 0x7ff5200a99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.60:41868/0 (socket says 172.21.15.60:41868) 2024-05-09T05:00:34.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7ff517fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff520096c10 0x7ff520201210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7ff517fff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff520096c10 0x7ff520201210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.60:35028/0 (socket says 172.21.15.60:35028) 2024-05-09T05:00:34.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7ff517fff640 1 -- 172.21.15.60:0/469954808 learned_addr learned my addr 172.21.15.60:0/469954808 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:34.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7fb0ebfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f4096d50 0x7fb0f40cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7fb0ebfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f4096d50 0x7fb0f40cf6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.60:35016/0 (socket says 172.21.15.60:35016) 2024-05-09T05:00:34.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7fb0ebfff640 1 -- 172.21.15.60:0/540519774 learned_addr learned my addr 172.21.15.60:0/540519774 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:34.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7ff525431640 1 --2- 172.21.15.60:0/469954808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff52009b160 0x7ff5200b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7f8182ffd640 1 -- 172.21.15.60:0/3741007679 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817c0f8b80 con 0x7f817c054260 2024-05-09T05:00:34.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7ff525431640 1 -- 172.21.15.60:0/469954808 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff520073100 con 0x7ff52009b160 2024-05-09T05:00:34.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7ff525431640 1 -- 172.21.15.60:0/469954808 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff520072f80 con 0x7ff5200962f0 2024-05-09T05:00:34.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7ff525431640 1 -- 172.21.15.60:0/469954808 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff520073280 con 0x7ff520096c10 2024-05-09T05:00:34.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7f4bdffff640 1 --2- 172.21.15.60:0/2161943980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bd802e1c0 0x7f4bd804e7e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4be00520f0 tx=0x7f4be0075310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7f81837fe640 1 --2- 172.21.15.60:0/3741007679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f817c033920 0x7f817c196f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:00:34.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7f618db53640 1 -- 172.21.15.60:0/1887603497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6180054430 msgr2=0x7f61800f3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7f618db53640 1 --2- 172.21.15.60:0/1887603497 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6180054430 0x7f61800f3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7f618db53640 1 -- 172.21.15.60:0/1887603497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6180033920 msgr2=0x7f6180103120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7f618db53640 1 --2- 172.21.15.60:0/1887603497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6180033920 0x7f6180103120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7f618db53640 1 -- 172.21.15.60:0/1887603497 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6180189740 con 0x7f6180101010 2024-05-09T05:00:34.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7f666d4b4640 1 -- 172.21.15.60:0/3394702519 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6648001030 con 0x7f666816c5a0 2024-05-09T05:00:34.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7fb0ebfff640 1 -- 172.21.15.60:0/540519774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f4071eb0 msgr2=0x7fb0f4091a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7fb0ebfff640 1 --2- 172.21.15.60:0/540519774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0f4071eb0 0x7fb0f4091a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7fb0ebfff640 1 -- 172.21.15.60:0/540519774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0f4097e10 msgr2=0x7fb0f40aadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7fb0ebfff640 1 --2- 172.21.15.60:0/540519774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0f4097e10 0x7fb0f40aadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7fb0ebfff640 1 -- 172.21.15.60:0/540519774 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0f420a410 con 0x7fb0f4096d50 2024-05-09T05:00:34.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7ff524c30640 1 --2- 172.21.15.60:0/469954808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff52009b160 0x7ff5200b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7ff51ffff640 1 -- 172.21.15.60:0/469954808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff520096c10 msgr2=0x7ff520201210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7ff51ffff640 1 --2- 172.21.15.60:0/469954808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff520096c10 0x7ff520201210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7ff51ffff640 1 -- 172.21.15.60:0/469954808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff52009b160 msgr2=0x7ff5200b45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7ff51ffff640 1 --2- 172.21.15.60:0/469954808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff52009b160 0x7ff5200b45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7f4bcbfff640 1 -- 172.21.15.60:0/2161943980 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be00b58e0 con 0x7f4bd802e1c0 2024-05-09T05:00:34.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7ff51ffff640 1 -- 172.21.15.60:0/469954808 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5200a9f00 con 0x7ff5200962f0 2024-05-09T05:00:34.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7ff517fff640 1 --2- 172.21.15.60:0/469954808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff520096c10 0x7ff520201210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:00:34.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7ff524c30640 1 --2- 172.21.15.60:0/469954808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff52009b160 0x7ff5200b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:34.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7f4bcbfff640 1 -- 172.21.15.60:0/2161943980 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4be0075aa0 con 0x7f4bd802e1c0 2024-05-09T05:00:34.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7ff51ffff640 1 --2- 172.21.15.60:0/469954808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5200962f0 0x7ff5200a99c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff510032000 tx=0x7ff5100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7fb0ebfff640 1 --2- 172.21.15.60:0/540519774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f4096d50 0x7fb0f40cf6d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb0f000e9e0 tx=0x7fb0f000eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7ff144ff9640 1 -- 172.21.15.60:0/1638895201 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff138017070 con 0x7ff148071eb0 2024-05-09T05:00:34.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.677+0000 7ff14e493640 1 -- 172.21.15.60:0/1638895201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff14816b480 con 0x7ff148071eb0 2024-05-09T05:00:34.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.678+0000 7ff14e493640 1 -- 172.21.15.60:0/1638895201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1480cc980 con 0x7ff148071eb0 2024-05-09T05:00:34.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7ff14e493640 1 -- 172.21.15.60:0/1638895201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff148051040 con 0x7ff148071eb0 2024-05-09T05:00:34.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7f8182ffd640 1 --2- 172.21.15.60:0/3741007679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f817c054260 0x7f817c0f8450 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f818406b970 tx=0x7f81840763a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7fb4fffff640 1 --2- 172.21.15.60:0/1698444652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb500097e10 0x7fb5001f7130 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb500098220 tx=0x7fb4f800cad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7f618db53640 1 --2- 172.21.15.60:0/1887603497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6180101010 0x7f61801a2ce0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f618802e140 tx=0x7f618806a6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7ff51d7fa640 1 -- 172.21.15.60:0/469954808 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff510043020 con 0x7ff5200962f0 2024-05-09T05:00:34.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7ff51d7fa640 1 -- 172.21.15.60:0/469954808 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5100347d0 con 0x7ff5200962f0 2024-05-09T05:00:34.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7ff51d7fa640 1 -- 172.21.15.60:0/469954808 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff510005bb0 con 0x7ff5200962f0 2024-05-09T05:00:34.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7ff525431640 1 -- 172.21.15.60:0/469954808 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff52016c8c0 con 0x7ff5200962f0 2024-05-09T05:00:34.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7ff144ff9640 1 -- 172.21.15.60:0/1638895201 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff138005da0 con 0x7ff148071eb0 2024-05-09T05:00:34.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7fb0e97fa640 1 -- 172.21.15.60:0/540519774 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0017070 con 0x7fb0f4096d50 2024-05-09T05:00:34.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7ff144ff9640 1 -- 172.21.15.60:0/1638895201 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff138007270 con 0x7ff148071eb0 2024-05-09T05:00:34.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7fb0fadf0640 1 -- 172.21.15.60:0/540519774 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0f4091f90 con 0x7fb0f4096d50 2024-05-09T05:00:34.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7f664f7fe640 1 -- 172.21.15.60:0/3394702519 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f666001b070 con 0x7f666816c5a0 2024-05-09T05:00:34.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7fb4e7fff640 1 -- 172.21.15.60:0/1698444652 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8003b70 con 0x7fb500097e10 2024-05-09T05:00:34.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7fb0fadf0640 1 -- 172.21.15.60:0/540519774 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0f4093e40 con 0x7fb0f4096d50 2024-05-09T05:00:34.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.680+0000 7fb0e97fa640 1 -- 172.21.15.60:0/540519774 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f000c9d0 con 0x7fb0f4096d50 2024-05-09T05:00:34.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.680+0000 7fb0e97fa640 1 -- 172.21.15.60:0/540519774 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f000cda0 con 0x7fb0f4096d50 2024-05-09T05:00:34.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7f4bcbfff640 1 -- 172.21.15.60:0/2161943980 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be00b5a80 con 0x7f4bd802e1c0 2024-05-09T05:00:34.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7f4be49f7640 1 -- 172.21.15.60:0/2161943980 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd802b700 con 0x7f4bd802e1c0 2024-05-09T05:00:34.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7f4be49f7640 1 -- 172.21.15.60:0/2161943980 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd80f3400 con 0x7f4bd802e1c0 2024-05-09T05:00:34.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.679+0000 7ff525431640 1 -- 172.21.15.60:0/469954808 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff52016ce80 con 0x7ff5200962f0 2024-05-09T05:00:34.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.680+0000 7f617dffb640 1 -- 172.21.15.60:0/1887603497 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6188051040 con 0x7f6180101010 2024-05-09T05:00:34.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.680+0000 7fb50617f640 1 -- 172.21.15.60:0/1698444652 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5001f53a0 con 0x7fb500097e10 2024-05-09T05:00:34.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.680+0000 7fb50617f640 1 -- 172.21.15.60:0/1698444652 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5001f5960 con 0x7fb500097e10 2024-05-09T05:00:34.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.680+0000 7f664f7fe640 1 --2- 172.21.15.60:0/3394702519 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f66540763a0 0x7f6654078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.680+0000 7f664f7fe640 1 -- 172.21.15.60:0/3394702519 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6660096f60 con 0x7f666816c5a0 2024-05-09T05:00:34.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.680+0000 7f664f7fe640 1 --2- 172.21.15.60:0/3394702519 >> [v2:172.21.15.114:6800/4142062928,v1:172.21.15.114:6802/4142062928] conn(0x7f665407fb10 0x7f6654081f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.681+0000 7f664f7fe640 1 -- 172.21.15.60:0/3394702519 --> [v2:172.21.15.114:6800/4142062928,v1:172.21.15.114:6802/4142062928] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6654082620 con 0x7f665407fb10 2024-05-09T05:00:34.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.680+0000 7fb4e7fff640 1 -- 172.21.15.60:0/1698444652 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4f8003d30 con 0x7fb500097e10 2024-05-09T05:00:34.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.681+0000 7f664f7fe640 1 -- 172.21.15.60:0/3394702519 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f666009c550 con 0x7f666816c5a0 2024-05-09T05:00:34.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.680+0000 7f816ffff640 1 -- 172.21.15.60:0/3741007679 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8184097020 con 0x7f817c054260 2024-05-09T05:00:34.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.680+0000 7f8188ce4640 1 -- 172.21.15.60:0/3741007679 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f817c0f8e60 con 0x7f817c054260 2024-05-09T05:00:34.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.681+0000 7fb4e7fff640 1 -- 172.21.15.60:0/1698444652 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8005310 con 0x7fb500097e10 2024-05-09T05:00:34.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.680+0000 7f8188ce4640 1 -- 172.21.15.60:0/3741007679 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f817c0f9420 con 0x7f817c054260 2024-05-09T05:00:34.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.680+0000 7f816ffff640 1 -- 172.21.15.60:0/3741007679 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f818407e8f0 con 0x7f817c054260 2024-05-09T05:00:34.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.680+0000 7f816ffff640 1 -- 172.21.15.60:0/3741007679 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8184097020 con 0x7f817c054260 2024-05-09T05:00:34.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.681+0000 7fb50617f640 1 -- 172.21.15.60:0/1698444652 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb4e0001030 con 0x7fb500097e10 2024-05-09T05:00:34.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.681+0000 7f618e354640 1 -- 172.21.15.60:0/1887603497 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61801a3850 con 0x7f6180101010 2024-05-09T05:00:34.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.681+0000 7f618e354640 1 -- 172.21.15.60:0/1887603497 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61800f2850 con 0x7f6180101010 2024-05-09T05:00:34.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.682+0000 7ff525431640 1 -- 172.21.15.60:0/469954808 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff52016c670 con 0x7ff5200962f0 2024-05-09T05:00:34.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.682+0000 7f617dffb640 1 -- 172.21.15.60:0/1887603497 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61880008d0 con 0x7f6180101010 2024-05-09T05:00:34.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.682+0000 7f617dffb640 1 -- 172.21.15.60:0/1887603497 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6188075aa0 con 0x7f6180101010 2024-05-09T05:00:34.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.682+0000 7ff144ff9640 1 -- 172.21.15.60:0/1638895201 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff13801a070 con 0x7ff148071eb0 2024-05-09T05:00:34.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.682+0000 7f66667fc640 1 --2- 172.21.15.60:0/3394702519 >> [v2:172.21.15.114:6800/4142062928,v1:172.21.15.114:6802/4142062928] conn(0x7f665407fb10 0x7f6654081f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.683+0000 7fb0e97fa640 1 -- 172.21.15.60:0/540519774 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb0f001a070 con 0x7fb0f4096d50 2024-05-09T05:00:34.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.683+0000 7fb0fadf0640 1 -- 172.21.15.60:0/540519774 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb0d4001030 con 0x7fb0f4096d50 2024-05-09T05:00:34.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.682+0000 7f4be49f7640 1 -- 172.21.15.60:0/2161943980 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4bd8005390 con 0x7f4bd802e1c0 2024-05-09T05:00:34.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.683+0000 7f4bcbfff640 1 -- 172.21.15.60:0/2161943980 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4be00bc060 con 0x7f4bd802e1c0 2024-05-09T05:00:34.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.683+0000 7ff144ff9640 1 --2- 172.21.15.60:0/1638895201 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff134076320 0x7ff1340787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.683+0000 7ff1477fe640 1 --2- 172.21.15.60:0/1638895201 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff134076320 0x7ff1340787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.683+0000 7fb4e7fff640 1 -- 172.21.15.60:0/1698444652 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb4f80054b0 con 0x7fb500097e10 2024-05-09T05:00:34.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.682+0000 7f8188ce4640 1 -- 172.21.15.60:0/3741007679 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8164001030 con 0x7f817c054260 2024-05-09T05:00:34.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.683+0000 7f816ffff640 1 -- 172.21.15.60:0/3741007679 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f818409a070 con 0x7f817c054260 2024-05-09T05:00:34.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.683+0000 7f6666ffd640 1 --2- 172.21.15.60:0/3394702519 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f66540763a0 0x7f6654078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.683+0000 7f816ffff640 1 --2- 172.21.15.60:0/3741007679 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f816807ece0 0x7f81680811a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7f66667fc640 1 --2- 172.21.15.60:0/3394702519 >> [v2:172.21.15.114:6800/4142062928,v1:172.21.15.114:6802/4142062928] conn(0x7f665407fb10 0x7f6654081f50 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7ff144ff9640 1 -- 172.21.15.60:0/1638895201 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff138097380 con 0x7ff148071eb0 2024-05-09T05:00:34.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7f816ffff640 1 -- 172.21.15.60:0/3741007679 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f81841229a0 con 0x7f817c054260 2024-05-09T05:00:34.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7f618e354640 1 -- 172.21.15.60:0/1887603497 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6168001030 con 0x7f6180101010 2024-05-09T05:00:34.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7ff51d7fa640 1 -- 172.21.15.60:0/469954808 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff510046020 con 0x7ff5200962f0 2024-05-09T05:00:34.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7ff1477fe640 1 --2- 172.21.15.60:0/1638895201 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff134076320 0x7ff1340787e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff13c006fd0 tx=0x7ff13c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7f816ffff640 1 --2- 172.21.15.60:0/3741007679 >> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] conn(0x7f81680883e0 0x7f816808a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7f4bcbfff640 1 --2- 172.21.15.60:0/2161943980 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4bc40763a0 0x7f4bc4078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7fb4e7fff640 1 --2- 172.21.15.60:0/1698444652 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb4ec0763a0 0x7fb4ec078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7ff144ff9640 1 --2- 172.21.15.60:0/1638895201 >> [v2:172.21.15.60:6816/483347218,v1:172.21.15.60:6817/483347218] conn(0x7ff13407faa0 0x7ff134081ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7f4bdf7fe640 1 --2- 172.21.15.60:0/2161943980 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4bc40763a0 0x7f4bc4078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7ff144ff9640 1 -- 172.21.15.60:0/1638895201 --> [v2:172.21.15.60:6816/483347218,v1:172.21.15.60:6817/483347218] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff138016ec0 con 0x7ff13407faa0 2024-05-09T05:00:34.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7ff144ff9640 1 -- 172.21.15.60:0/1638895201 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff138097730 con 0x7ff148071eb0 2024-05-09T05:00:34.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7fb4ff7fe640 1 --2- 172.21.15.60:0/1698444652 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb4ec0763a0 0x7fb4ec078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7f816ffff640 1 -- 172.21.15.60:0/3741007679 --> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8184080ea0 con 0x7f81680883e0 2024-05-09T05:00:34.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7f816ffff640 1 -- 172.21.15.60:0/3741007679 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f81841272a0 con 0x7f817c054260 2024-05-09T05:00:34.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7f81837fe640 1 --2- 172.21.15.60:0/3741007679 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f816807ece0 0x7f81680811a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7f8183fff640 1 --2- 172.21.15.60:0/3741007679 >> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] conn(0x7f81680883e0 0x7f816808a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7f81837fe640 1 --2- 172.21.15.60:0/3741007679 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f816807ece0 0x7f81680811a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f817800bbd0 tx=0x7f817800d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7f4bcbfff640 1 -- 172.21.15.60:0/2161943980 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4be0101a50 con 0x7f4bd802e1c0 2024-05-09T05:00:34.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7fb4e7fff640 1 -- 172.21.15.60:0/1698444652 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb4f8098200 con 0x7fb500097e10 2024-05-09T05:00:34.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7f617dffb640 1 -- 172.21.15.60:0/1887603497 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6188075e60 con 0x7f6180101010 2024-05-09T05:00:34.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.684+0000 7ff14ca30640 1 --2- 172.21.15.60:0/1638895201 >> [v2:172.21.15.60:6816/483347218,v1:172.21.15.60:6817/483347218] conn(0x7ff13407faa0 0x7ff134081ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7f664f7fe640 1 -- 172.21.15.60:0/3394702519 <== osd.7 v2:172.21.15.114:6800/4142062928 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6654082620 con 0x7f665407fb10 2024-05-09T05:00:34.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7f6666ffd640 1 --2- 172.21.15.60:0/3394702519 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f66540763a0 0x7f6654078860 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6658033000 tx=0x7f665802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7f4bdf7fe640 1 --2- 172.21.15.60:0/2161943980 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4bc40763a0 0x7f4bc4078860 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4034000 tx=0x7f4bd402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7ff51d7fa640 1 --2- 172.21.15.60:0/469954808 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff5000762d0 0x7ff500078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7f4bcbfff640 1 --2- 172.21.15.60:0/2161943980 >> [v2:172.21.15.60:6802/291519275,v1:172.21.15.60:6804/291519275] conn(0x7f4bc407fb20 0x7f4bc4081f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7fb4ff7fe640 1 --2- 172.21.15.60:0/1698444652 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb4ec0763a0 0x7fb4ec078860 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb4f000b820 tx=0x7fb4f0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7f8183fff640 1 --2- 172.21.15.60:0/3741007679 >> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] conn(0x7f81680883e0 0x7f816808a820 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7f4bcbfff640 1 -- 172.21.15.60:0/2161943980 --> [v2:172.21.15.60:6802/291519275,v1:172.21.15.60:6804/291519275] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4be006ce50 con 0x7f4bc407fb20 2024-05-09T05:00:34.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7f4bcbfff640 1 -- 172.21.15.60:0/2161943980 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4be01062a0 con 0x7f4bd802e1c0 2024-05-09T05:00:34.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7fb4e7fff640 1 --2- 172.21.15.60:0/1698444652 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x7fb4ec07fb20 0x7fb4ec081f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7fb4e7fff640 1 -- 172.21.15.60:0/1698444652 --> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb4f803cec0 con 0x7fb4ec07fb20 2024-05-09T05:00:34.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7fb4e7fff640 1 -- 172.21.15.60:0/1698444652 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb4f80985f0 con 0x7fb500097e10 2024-05-09T05:00:34.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7fb0e97fa640 1 --2- 172.21.15.60:0/540519774 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb0e00763a0 0x7fb0e0078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7ff51d7fa640 1 -- 172.21.15.60:0/469954808 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff5100b87a0 con 0x7ff5200962f0 2024-05-09T05:00:34.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7fb0e97fa640 1 -- 172.21.15.60:0/540519774 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb0f00978a0 con 0x7fb0f4096d50 2024-05-09T05:00:34.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7fb0f8b8c640 1 --2- 172.21.15.60:0/540519774 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb0e00763a0 0x7fb0e0078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7fb0e97fa640 1 --2- 172.21.15.60:0/540519774 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x7fb0e007faa0 0x7fb0e0081ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7ff51d7fa640 1 --2- 172.21.15.60:0/469954808 >> [v2:172.21.15.114:6824/2612522523,v1:172.21.15.114:6825/2612522523] conn(0x7ff50007fa50 0x7ff500081e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7fb0e97fa640 1 -- 172.21.15.60:0/540519774 --> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb0e00825b0 con 0x7fb0e007faa0 2024-05-09T05:00:34.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7fb0e97fa640 1 -- 172.21.15.60:0/540519774 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb0f009e050 con 0x7fb0f4096d50 2024-05-09T05:00:34.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7ff517fff640 1 --2- 172.21.15.60:0/469954808 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff5000762d0 0x7ff500078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7ff51d7fa640 1 -- 172.21.15.60:0/469954808 --> [v2:172.21.15.114:6824/2612522523,v1:172.21.15.114:6825/2612522523] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff51003ee50 con 0x7ff50007fa50 2024-05-09T05:00:34.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7ff51d7fa640 1 -- 172.21.15.60:0/469954808 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff5100bf050 con 0x7ff5200962f0 2024-05-09T05:00:34.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.685+0000 7f617dffb640 1 --2- 172.21.15.60:0/1887603497 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f616c076300 0x7f616c0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.686+0000 7f617dffb640 1 -- 172.21.15.60:0/1887603497 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6188102500 con 0x7f6180101010 2024-05-09T05:00:34.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.686+0000 7fb4feffd640 1 --2- 172.21.15.60:0/1698444652 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x7fb4ec07fb20 0x7fb4ec081f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.686+0000 7fb0f938d640 1 --2- 172.21.15.60:0/540519774 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x7fb0e007faa0 0x7fb0e0081ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.686+0000 7ff524c30640 1 --2- 172.21.15.60:0/469954808 >> [v2:172.21.15.114:6824/2612522523,v1:172.21.15.114:6825/2612522523] conn(0x7ff50007fa50 0x7ff500081e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.686+0000 7ff14ca30640 1 --2- 172.21.15.60:0/1638895201 >> [v2:172.21.15.60:6816/483347218,v1:172.21.15.60:6817/483347218] conn(0x7ff13407faa0 0x7ff134081ee0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.686+0000 7f816ffff640 1 -- 172.21.15.60:0/3741007679 <== osd.5 v2:172.21.15.114:6816/292113074 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8184080ea0 con 0x7f81680883e0 2024-05-09T05:00:34.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.686+0000 7fb0f8b8c640 1 --2- 172.21.15.60:0/540519774 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb0e00763a0 0x7fb0e0078860 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4001220 tx=0x7fb0e4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.686+0000 7f618d352640 1 --2- 172.21.15.60:0/1887603497 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f616c076300 0x7f616c0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.686+0000 7f617dffb640 1 --2- 172.21.15.60:0/1887603497 >> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] conn(0x7f616c07fa70 0x7f616c081eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:34.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.686+0000 7f617dffb640 1 -- 172.21.15.60:0/1887603497 --> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f616c082580 con 0x7f616c07fa70 2024-05-09T05:00:34.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.686+0000 7fb4feffd640 1 --2- 172.21.15.60:0/1698444652 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x7fb4ec07fb20 0x7fb4ec081f60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.686+0000 7ff517fff640 1 --2- 172.21.15.60:0/469954808 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff5000762d0 0x7ff500078790 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff5200b52d0 tx=0x7ff50800b670 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.686+0000 7f617dffb640 1 -- 172.21.15.60:0/1887603497 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6188109050 con 0x7f6180101010 2024-05-09T05:00:34.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.686+0000 7f618cb51640 1 --2- 172.21.15.60:0/1887603497 >> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] conn(0x7f616c07fa70 0x7f616c081eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.687+0000 7f618d352640 1 --2- 172.21.15.60:0/1887603497 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f616c076300 0x7f616c0787c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6178034000 tx=0x7f6178009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.686+0000 7f4bdeffd640 1 --2- 172.21.15.60:0/2161943980 >> [v2:172.21.15.60:6802/291519275,v1:172.21.15.60:6804/291519275] conn(0x7f4bc407fb20 0x7f4bc4081f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:34.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.687+0000 7f4bdeffd640 1 --2- 172.21.15.60:0/2161943980 >> [v2:172.21.15.60:6802/291519275,v1:172.21.15.60:6804/291519275] conn(0x7f4bc407fb20 0x7f4bc4081f60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.687+0000 7fb0f938d640 1 --2- 172.21.15.60:0/540519774 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x7fb0e007faa0 0x7fb0e0081ee0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.776 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.687+0000 7ff144ff9640 1 -- 172.21.15.60:0/1638895201 <== osd.2 v2:172.21.15.60:6816/483347218 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff138016ec0 con 0x7ff13407faa0 2024-05-09T05:00:34.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.687+0000 7fb4e7fff640 1 -- 172.21.15.60:0/1698444652 <== osd.3 v2:172.21.15.60:6824/1069806621 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb4f803cec0 con 0x7fb4ec07fb20 2024-05-09T05:00:34.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.687+0000 7ff524c30640 1 --2- 172.21.15.60:0/469954808 >> [v2:172.21.15.114:6824/2612522523,v1:172.21.15.114:6825/2612522523] conn(0x7ff50007fa50 0x7ff500081e90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.687+0000 7f618cb51640 1 --2- 172.21.15.60:0/1887603497 >> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] conn(0x7f616c07fa70 0x7f616c081eb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:34.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.688+0000 7f4bcbfff640 1 -- 172.21.15.60:0/2161943980 <== osd.1 v2:172.21.15.60:6802/291519275 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4be006ce50 con 0x7f4bc407fb20 2024-05-09T05:00:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.688+0000 7ff51d7fa640 1 -- 172.21.15.60:0/469954808 <== osd.6 v2:172.21.15.114:6824/2612522523 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff51003ee50 con 0x7ff50007fa50 2024-05-09T05:00:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.688+0000 7f617dffb640 1 -- 172.21.15.60:0/1887603497 <== osd.4 v2:172.21.15.114:6801/3462725972 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f616c082580 con 0x7f616c07fa70 2024-05-09T05:00:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.688+0000 7fb0e97fa640 1 -- 172.21.15.60:0/540519774 <== osd.0 v2:172.21.15.60:6800/1298086532 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb0e00825b0 con 0x7fb0e007faa0 2024-05-09T05:00:34.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.715+0000 7f666d4b4640 1 -- 172.21.15.60:0/3394702519 --> [v2:172.21.15.114:6800/4142062928,v1:172.21.15.114:6802/4142062928] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6648002dd0 con 0x7f665407fb10 2024-05-09T05:00:34.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.717+0000 7f664f7fe640 1 -- 172.21.15.60:0/3394702519 <== osd.7 v2:172.21.15.114:6800/4142062928 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6648002dd0 con 0x7f665407fb10 2024-05-09T05:00:34.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.718+0000 7ff525431640 1 -- 172.21.15.60:0/469954808 --> [v2:172.21.15.114:6824/2612522523,v1:172.21.15.114:6825/2612522523] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff5200640c0 con 0x7ff50007fa50 2024-05-09T05:00:34.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.718+0000 7f666d4b4640 1 -- 172.21.15.60:0/3394702519 >> [v2:172.21.15.114:6800/4142062928,v1:172.21.15.114:6802/4142062928] conn(0x7f665407fb10 msgr2=0x7f6654081f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.718+0000 7f666d4b4640 1 --2- 172.21.15.60:0/3394702519 >> [v2:172.21.15.114:6800/4142062928,v1:172.21.15.114:6802/4142062928] conn(0x7f665407fb10 0x7f6654081f50 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.718+0000 7f666d4b4640 1 -- 172.21.15.60:0/3394702519 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f66540763a0 msgr2=0x7f6654078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.718+0000 7f666d4b4640 1 --2- 172.21.15.60:0/3394702519 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f66540763a0 0x7f6654078860 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6658033000 tx=0x7f665802e000 comp rx=0 tx=0).stop 2024-05-09T05:00:34.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.718+0000 7f666d4b4640 1 -- 172.21.15.60:0/3394702519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f666816c5a0 msgr2=0x7f666820add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.718+0000 7f666d4b4640 1 --2- 172.21.15.60:0/3394702519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f666816c5a0 0x7f666820add0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f666000c7a0 tx=0x7f666000cc60 comp rx=0 tx=0).stop 2024-05-09T05:00:34.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.718+0000 7f66677fe640 1 -- 172.21.15.60:0/3394702519 reap_dead start 2024-05-09T05:00:34.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.718+0000 7f666d4b4640 1 -- 172.21.15.60:0/3394702519 shutdown_connections 2024-05-09T05:00:34.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.719+0000 7ff51d7fa640 1 -- 172.21.15.60:0/469954808 <== osd.6 v2:172.21.15.114:6824/2612522523 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff5200640c0 con 0x7ff50007fa50 2024-05-09T05:00:34.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.718+0000 7f666d4b4640 1 -- 172.21.15.60:0/3394702519 >> 172.21.15.60:0/3394702519 conn(0x7f66680afbd0 msgr2=0x7f6668099fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:34.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.719+0000 7f666d4b4640 1 -- 172.21.15.60:0/3394702519 shutdown_connections 2024-05-09T05:00:34.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.719+0000 7f666d4b4640 1 -- 172.21.15.60:0/3394702519 wait complete. 2024-05-09T05:00:34.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.719+0000 7ff525431640 1 -- 172.21.15.60:0/469954808 >> [v2:172.21.15.114:6824/2612522523,v1:172.21.15.114:6825/2612522523] conn(0x7ff50007fa50 msgr2=0x7ff500081e90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.719+0000 7ff525431640 1 --2- 172.21.15.60:0/469954808 >> [v2:172.21.15.114:6824/2612522523,v1:172.21.15.114:6825/2612522523] conn(0x7ff50007fa50 0x7ff500081e90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.719+0000 7ff525431640 1 -- 172.21.15.60:0/469954808 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff5000762d0 msgr2=0x7ff500078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.719+0000 7ff525431640 1 --2- 172.21.15.60:0/469954808 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff5000762d0 0x7ff500078790 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff5200b52d0 tx=0x7ff50800b670 comp rx=0 tx=0).stop 2024-05-09T05:00:34.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.719+0000 7ff525431640 1 -- 172.21.15.60:0/469954808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5200962f0 msgr2=0x7ff5200a99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.719+0000 7ff525431640 1 --2- 172.21.15.60:0/469954808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5200962f0 0x7ff5200a99c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff510032000 tx=0x7ff5100046c0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.720+0000 7ff524c30640 1 -- 172.21.15.60:0/469954808 reap_dead start 2024-05-09T05:00:34.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.720+0000 7ff525431640 1 -- 172.21.15.60:0/469954808 shutdown_connections 2024-05-09T05:00:34.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.720+0000 7ff525431640 1 -- 172.21.15.60:0/469954808 >> 172.21.15.60:0/469954808 conn(0x7ff5200b0940 msgr2=0x7ff52009a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:34.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.720+0000 7ff525431640 1 -- 172.21.15.60:0/469954808 shutdown_connections 2024-05-09T05:00:34.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.720+0000 7ff525431640 1 -- 172.21.15.60:0/469954808 wait complete. 2024-05-09T05:00:34.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.720+0000 7f8188ce4640 1 -- 172.21.15.60:0/3741007679 --> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8164002dd0 con 0x7f81680883e0 2024-05-09T05:00:34.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.721+0000 7fb50617f640 1 -- 172.21.15.60:0/1698444652 --> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb4e0002dd0 con 0x7fb4ec07fb20 2024-05-09T05:00:34.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.721+0000 7f816ffff640 1 -- 172.21.15.60:0/3741007679 <== osd.5 v2:172.21.15.114:6816/292113074 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8164002dd0 con 0x7f81680883e0 2024-05-09T05:00:34.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.721+0000 7ff14e493640 1 -- 172.21.15.60:0/1638895201 --> [v2:172.21.15.60:6816/483347218,v1:172.21.15.60:6817/483347218] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff148093e90 con 0x7ff13407faa0 2024-05-09T05:00:34.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.721+0000 7fb4e7fff640 1 -- 172.21.15.60:0/1698444652 <== osd.3 v2:172.21.15.60:6824/1069806621 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb4e0002dd0 con 0x7fb4ec07fb20 2024-05-09T05:00:34.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.722+0000 7ff144ff9640 1 -- 172.21.15.60:0/1638895201 <== osd.2 v2:172.21.15.60:6816/483347218 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff148093e90 con 0x7ff13407faa0 2024-05-09T05:00:34.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.722+0000 7f816dffb640 1 -- 172.21.15.60:0/3741007679 >> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] conn(0x7f81680883e0 msgr2=0x7f816808a820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.722+0000 7f816dffb640 1 --2- 172.21.15.60:0/3741007679 >> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] conn(0x7f81680883e0 0x7f816808a820 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.722+0000 7fb50617f640 1 -- 172.21.15.60:0/1698444652 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x7fb4ec07fb20 msgr2=0x7fb4ec081f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.722+0000 7fb50617f640 1 --2- 172.21.15.60:0/1698444652 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x7fb4ec07fb20 0x7fb4ec081f60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.722+0000 7ff12e7fc640 1 -- 172.21.15.60:0/1638895201 >> [v2:172.21.15.60:6816/483347218,v1:172.21.15.60:6817/483347218] conn(0x7ff13407faa0 msgr2=0x7ff134081ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.722+0000 7ff12e7fc640 1 --2- 172.21.15.60:0/1638895201 >> [v2:172.21.15.60:6816/483347218,v1:172.21.15.60:6817/483347218] conn(0x7ff13407faa0 0x7ff134081ee0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.722+0000 7fb50617f640 1 -- 172.21.15.60:0/1698444652 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb4ec0763a0 msgr2=0x7fb4ec078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.722+0000 7fb50617f640 1 --2- 172.21.15.60:0/1698444652 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb4ec0763a0 0x7fb4ec078860 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb4f000b820 tx=0x7fb4f0030000 comp rx=0 tx=0).stop 2024-05-09T05:00:34.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.722+0000 7fb50617f640 1 -- 172.21.15.60:0/1698444652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb500097e10 msgr2=0x7fb5001f7130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.722+0000 7fb50617f640 1 --2- 172.21.15.60:0/1698444652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb500097e10 0x7fb5001f7130 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb500098220 tx=0x7fb4f800cad0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.722+0000 7fb4fffff640 1 -- 172.21.15.60:0/1698444652 reap_dead start 2024-05-09T05:00:34.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.722+0000 7f618e354640 1 -- 172.21.15.60:0/1887603497 --> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6168002dd0 con 0x7f616c07fa70 2024-05-09T05:00:34.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.722+0000 7f816dffb640 1 -- 172.21.15.60:0/3741007679 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f816807ece0 msgr2=0x7f81680811a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.722+0000 7f816dffb640 1 --2- 172.21.15.60:0/3741007679 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f816807ece0 0x7f81680811a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f817800bbd0 tx=0x7f817800d040 comp rx=0 tx=0).stop 2024-05-09T05:00:34.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.722+0000 7fb0fadf0640 1 -- 172.21.15.60:0/540519774 --> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb0d4002dd0 con 0x7fb0e007faa0 2024-05-09T05:00:34.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7f816dffb640 1 -- 172.21.15.60:0/3741007679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f817c054260 msgr2=0x7f817c0f8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7f816dffb640 1 --2- 172.21.15.60:0/3741007679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f817c054260 0x7f817c0f8450 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f818406b970 tx=0x7f81840763a0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7f8183fff640 1 -- 172.21.15.60:0/3741007679 reap_dead start 2024-05-09T05:00:34.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7ff12e7fc640 1 -- 172.21.15.60:0/1638895201 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff134076320 msgr2=0x7ff1340787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7ff12e7fc640 1 --2- 172.21.15.60:0/1638895201 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff134076320 0x7ff1340787e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff13c006fd0 tx=0x7ff13c008040 comp rx=0 tx=0).stop 2024-05-09T05:00:34.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7ff12e7fc640 1 -- 172.21.15.60:0/1638895201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff148071eb0 msgr2=0x7ff148093950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7ff12e7fc640 1 --2- 172.21.15.60:0/1638895201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff148071eb0 0x7ff148093950 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff13800c500 tx=0x7ff13800c9c0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7ff14ca30640 1 -- 172.21.15.60:0/1638895201 reap_dead start 2024-05-09T05:00:34.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7f816dffb640 1 -- 172.21.15.60:0/3741007679 shutdown_connections 2024-05-09T05:00:34.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7f816dffb640 1 -- 172.21.15.60:0/3741007679 >> 172.21.15.60:0/3741007679 conn(0x7f817c009fb0 msgr2=0x7f817c036410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:34.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7ff12e7fc640 1 -- 172.21.15.60:0/1638895201 shutdown_connections 2024-05-09T05:00:34.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7ff12e7fc640 1 -- 172.21.15.60:0/1638895201 >> 172.21.15.60:0/1638895201 conn(0x7ff1480972f0 msgr2=0x7ff1480976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:34.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7fb50617f640 1 -- 172.21.15.60:0/1698444652 shutdown_connections 2024-05-09T05:00:34.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7fb50617f640 1 -- 172.21.15.60:0/1698444652 >> 172.21.15.60:0/1698444652 conn(0x7fb5000972f0 msgr2=0x7fb50006eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:34.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7f617dffb640 1 -- 172.21.15.60:0/1887603497 <== osd.4 v2:172.21.15.114:6801/3462725972 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6168002dd0 con 0x7f616c07fa70 2024-05-09T05:00:34.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7fb0e97fa640 1 -- 172.21.15.60:0/540519774 <== osd.0 v2:172.21.15.60:6800/1298086532 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb0d4002dd0 con 0x7fb0e007faa0 2024-05-09T05:00:34.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7f816dffb640 1 -- 172.21.15.60:0/3741007679 shutdown_connections 2024-05-09T05:00:34.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7f816dffb640 1 -- 172.21.15.60:0/3741007679 wait complete. 2024-05-09T05:00:34.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7ff12e7fc640 1 -- 172.21.15.60:0/1638895201 shutdown_connections 2024-05-09T05:00:34.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7ff12e7fc640 1 -- 172.21.15.60:0/1638895201 wait complete. 2024-05-09T05:00:34.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7fb0daffd640 1 -- 172.21.15.60:0/540519774 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x7fb0e007faa0 msgr2=0x7fb0e0081ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7fb0daffd640 1 --2- 172.21.15.60:0/540519774 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x7fb0e007faa0 0x7fb0e0081ee0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.723+0000 7f4be49f7640 1 -- 172.21.15.60:0/2161943980 --> [v2:172.21.15.60:6802/291519275,v1:172.21.15.60:6804/291519275] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4bd81024c0 con 0x7f4bc407fb20 2024-05-09T05:00:34.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7f61737fe640 1 -- 172.21.15.60:0/1887603497 >> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] conn(0x7f616c07fa70 msgr2=0x7f616c081eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7f61737fe640 1 --2- 172.21.15.60:0/1887603497 >> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] conn(0x7f616c07fa70 0x7f616c081eb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7fb0daffd640 1 -- 172.21.15.60:0/540519774 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb0e00763a0 msgr2=0x7fb0e0078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7fb0daffd640 1 --2- 172.21.15.60:0/540519774 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb0e00763a0 0x7fb0e0078860 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4001220 tx=0x7fb0e4009040 comp rx=0 tx=0).stop 2024-05-09T05:00:34.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7fb0daffd640 1 -- 172.21.15.60:0/540519774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f4096d50 msgr2=0x7fb0f40cf6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7fb0daffd640 1 --2- 172.21.15.60:0/540519774 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb0f4096d50 0x7fb0f40cf6d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb0f000e9e0 tx=0x7fb0f000eea0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7fb50617f640 1 -- 172.21.15.60:0/1698444652 shutdown_connections 2024-05-09T05:00:34.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7f61737fe640 1 -- 172.21.15.60:0/1887603497 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f616c076300 msgr2=0x7f616c0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7f61737fe640 1 --2- 172.21.15.60:0/1887603497 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f616c076300 0x7f616c0787c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6178034000 tx=0x7f6178009040 comp rx=0 tx=0).stop 2024-05-09T05:00:34.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7f61737fe640 1 -- 172.21.15.60:0/1887603497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6180101010 msgr2=0x7f61801a2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7f61737fe640 1 --2- 172.21.15.60:0/1887603497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6180101010 0x7f61801a2ce0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f618802e140 tx=0x7f618806a6a0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7fb0f938d640 1 -- 172.21.15.60:0/540519774 reap_dead start 2024-05-09T05:00:34.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7f4bcbfff640 1 -- 172.21.15.60:0/2161943980 <== osd.1 v2:172.21.15.60:6802/291519275 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4bd81024c0 con 0x7f4bc407fb20 2024-05-09T05:00:34.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7fb50617f640 1 -- 172.21.15.60:0/1698444652 wait complete. 2024-05-09T05:00:34.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7f618db53640 1 -- 172.21.15.60:0/1887603497 reap_dead start 2024-05-09T05:00:34.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7f61737fe640 1 -- 172.21.15.60:0/1887603497 shutdown_connections 2024-05-09T05:00:34.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7f61737fe640 1 -- 172.21.15.60:0/1887603497 >> 172.21.15.60:0/1887603497 conn(0x7f6180009fb0 msgr2=0x7f6180035380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:34.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7fb0daffd640 1 -- 172.21.15.60:0/540519774 shutdown_connections 2024-05-09T05:00:34.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7fb0daffd640 1 -- 172.21.15.60:0/540519774 >> 172.21.15.60:0/540519774 conn(0x7fb0f40972f0 msgr2=0x7fb0f406ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:34.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.724+0000 7f61737fe640 1 -- 172.21.15.60:0/1887603497 shutdown_connections 2024-05-09T05:00:34.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.725+0000 7f61737fe640 1 -- 172.21.15.60:0/1887603497 wait complete. 2024-05-09T05:00:34.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.725+0000 7fb0daffd640 1 -- 172.21.15.60:0/540519774 shutdown_connections 2024-05-09T05:00:34.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.725+0000 7f4be49f7640 1 -- 172.21.15.60:0/2161943980 >> [v2:172.21.15.60:6802/291519275,v1:172.21.15.60:6804/291519275] conn(0x7f4bc407fb20 msgr2=0x7f4bc4081f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.725+0000 7fb0daffd640 1 -- 172.21.15.60:0/540519774 wait complete. 2024-05-09T05:00:34.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.725+0000 7f4be49f7640 1 --2- 172.21.15.60:0/2161943980 >> [v2:172.21.15.60:6802/291519275,v1:172.21.15.60:6804/291519275] conn(0x7f4bc407fb20 0x7f4bc4081f60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:34.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.725+0000 7f4be49f7640 1 -- 172.21.15.60:0/2161943980 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4bc40763a0 msgr2=0x7f4bc4078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.725+0000 7f4be49f7640 1 --2- 172.21.15.60:0/2161943980 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4bc40763a0 0x7f4bc4078860 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4034000 tx=0x7f4bd402e000 comp rx=0 tx=0).stop 2024-05-09T05:00:34.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.725+0000 7f4be49f7640 1 -- 172.21.15.60:0/2161943980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bd802e1c0 msgr2=0x7f4bd804e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:34.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.725+0000 7f4be49f7640 1 --2- 172.21.15.60:0/2161943980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bd802e1c0 0x7f4bd804e7e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4be00520f0 tx=0x7f4be0075310 comp rx=0 tx=0).stop 2024-05-09T05:00:34.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.725+0000 7f4bdffff640 1 -- 172.21.15.60:0/2161943980 reap_dead start 2024-05-09T05:00:34.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.725+0000 7f4be49f7640 1 -- 172.21.15.60:0/2161943980 shutdown_connections 2024-05-09T05:00:34.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.725+0000 7f4be49f7640 1 -- 172.21.15.60:0/2161943980 >> 172.21.15.60:0/2161943980 conn(0x7f4bd8009fb0 msgr2=0x7f4bd8102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:34.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.726+0000 7f4be49f7640 1 -- 172.21.15.60:0/2161943980 shutdown_connections 2024-05-09T05:00:34.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:34.726+0000 7f4be49f7640 1 -- 172.21.15.60:0/2161943980 wait complete. 2024-05-09T05:00:34.799 INFO:teuthology.orchestra.run.smithi060.stdout:55834574855 2024-05-09T05:00:34.800 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-09T05:00:34.800 INFO:teuthology.orchestra.run.smithi060.stdout:55834574856 2024-05-09T05:00:34.801 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-09T05:00:34.801 INFO:teuthology.orchestra.run.smithi060.stdout:55834574856 2024-05-09T05:00:34.802 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-09T05:00:34.802 INFO:teuthology.orchestra.run.smithi060.stdout:55834574855 2024-05-09T05:00:34.803 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-09T05:00:34.803 INFO:teuthology.orchestra.run.smithi060.stdout:55834574855 2024-05-09T05:00:34.804 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-09T05:00:34.805 INFO:teuthology.orchestra.run.smithi060.stdout:55834574856 2024-05-09T05:00:34.805 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-09T05:00:34.806 INFO:teuthology.orchestra.run.smithi060.stdout:55834574855 2024-05-09T05:00:34.806 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-09T05:00:34.807 INFO:teuthology.orchestra.run.smithi060.stdout:55834574855 2024-05-09T05:00:34.807 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-09T05:00:35.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.020+0000 7fad6530a640 1 Processor -- start 2024-05-09T05:00:35.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.020+0000 7f8872d78640 1 Processor -- start 2024-05-09T05:00:35.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.020+0000 7f8872d78640 1 -- start start 2024-05-09T05:00:35.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.021+0000 7fb40759e640 1 Processor -- start 2024-05-09T05:00:35.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.022+0000 7fb40759e640 1 -- start start 2024-05-09T05:00:35.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.022+0000 7f8872d78640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f886c0d0900 0x7f886c15c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.022+0000 7f8872d78640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f886c15b520 0x7f886c073790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.022+0000 7f8872d78640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f886c0985e0 con 0x7f886c06c670 2024-05-09T05:00:35.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.022+0000 7f8872d78640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f886c15c950 con 0x7f886c15b520 2024-05-09T05:00:35.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.022+0000 7f8872d78640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f886c0991f0 con 0x7f886c0d0900 2024-05-09T05:00:35.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.022+0000 7f8872577640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f886c15b520 0x7f886c073790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.022+0000 7f8872577640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f886c15b520 0x7f886c073790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34942/0 (socket says 172.21.15.60:34942) 2024-05-09T05:00:35.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.022+0000 7f8872577640 1 -- 172.21.15.60:0/209166590 learned_addr learned my addr 172.21.15.60:0/209166590 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:35.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.020+0000 7fad6530a640 1 -- start start 2024-05-09T05:00:35.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.022+0000 7fad6530a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fad580dfcb0 0x7fad580e9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.022+0000 7fad6530a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad580e0090 con 0x7fad58054690 2024-05-09T05:00:35.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.022+0000 7fad6530a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad58053310 con 0x7fad580df1c0 2024-05-09T05:00:35.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.023+0000 7f8872577640 1 -- 172.21.15.60:0/209166590 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f886c0d0900 msgr2=0x7f886c15c410 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T05:00:35.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.023+0000 7f8872577640 1 --2- 172.21.15.60:0/209166590 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f886c0d0900 0x7f886c15c410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.023+0000 7f8872577640 1 -- 172.21.15.60:0/209166590 >> v1:172.21.15.114:6789/0 conn(0x7f886c06c670 legacy=0x7f886c0d01f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T05:00:35.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.023+0000 7f8872577640 1 -- 172.21.15.60:0/209166590 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886c0988c0 con 0x7f886c15b520 2024-05-09T05:00:35.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.022+0000 7fad6530a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fad580535f0 con 0x7fad580dfcb0 2024-05-09T05:00:35.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.022+0000 7fad5e7fc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fad58054690 0x7fad580df5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38466/0 (socket says 172.21.15.60:38466) 2024-05-09T05:00:35.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.022+0000 7fad5e7fc640 1 -- 172.21.15.60:0/2628339518 learned_addr learned my addr 172.21.15.60:0/2628339518 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:35.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.023+0000 7fad5f7fe640 1 --2- 172.21.15.60:0/2628339518 >> v2:172.21.15.60:3300/0 conn(0x7fad580dfcb0 0x7fad580e9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.023+0000 7fb40759e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb40806fcb0 0x7fb408073ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.023+0000 7fb40759e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb4080ab290 0x7fb40816d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.023+0000 7fb40759e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4080a91e0 con 0x7fb408073040 2024-05-09T05:00:35.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.023+0000 7fb40759e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fb4080a94c0 con 0x7fb4080ab290 2024-05-09T05:00:35.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.023+0000 7fb40759e640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fb4080a97a0 con 0x7fb40806fcb0 2024-05-09T05:00:35.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.024+0000 7fb406d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb4080ab290 0x7fb40816d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.024+0000 7fb405d9b640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb408073040 0x7fb408070090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.60:45282/0 (socket says 172.21.15.60:45282) 2024-05-09T05:00:35.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.024+0000 7fb406d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb4080ab290 0x7fb40816d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34972/0 (socket says 172.21.15.60:34972) 2024-05-09T05:00:35.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.024+0000 7fb405d9b640 1 -- 172.21.15.60:0/2641843161 learned_addr learned my addr 172.21.15.60:0/2641843161 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:35.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.024+0000 7fb40659c640 1 --2- 172.21.15.60:0/2641843161 >> v2:172.21.15.114:3300/0 conn(0x7fb40806fcb0 0x7fb408073ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.024+0000 7f8872577640 1 --2- 172.21.15.60:0/209166590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886c15b520 0x7f886c073790 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f886400b1a0 tx=0x7f886402a980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6de06c091a298e8d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.024+0000 7fb406d9d640 1 -- 172.21.15.60:0/2641843161 >> v2:172.21.15.114:3300/0 conn(0x7fb40806fcb0 msgr2=0x7fb408073ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:35.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.024+0000 7fb406d9d640 1 --2- 172.21.15.60:0/2641843161 >> v2:172.21.15.114:3300/0 conn(0x7fb40806fcb0 0x7fb408073ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.024+0000 7fb40559a640 1 -- 172.21.15.60:0/2641843161 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1709482322 0 0) 0x7fb4080a91e0 con 0x7fb408073040 2024-05-09T05:00:35.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.024+0000 7fb406d9d640 1 -- 172.21.15.60:0/2641843161 >> v1:172.21.15.114:6789/0 conn(0x7fb408073040 legacy=0x7fb408070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.024+0000 7fb406d9d640 1 -- 172.21.15.60:0/2641843161 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4080a9a80 con 0x7fb4080ab290 2024-05-09T05:00:35.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.024+0000 7fb40659c640 1 --2- 172.21.15.60:0/2641843161 >> v2:172.21.15.114:3300/0 conn(0x7fb40806fcb0 0x7fb408073ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:35.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.025+0000 7fad5f7fe640 1 -- 172.21.15.60:0/2628339518 >> v1:172.21.15.114:6789/0 conn(0x7fad580df1c0 legacy=0x7fad5804fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.025+0000 7f8870d74640 1 -- 172.21.15.60:0/209166590 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8864030470 con 0x7f886c15b520 2024-05-09T05:00:35.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.025+0000 7f8870d74640 1 -- 172.21.15.60:0/209166590 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8864035070 con 0x7f886c15b520 2024-05-09T05:00:35.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.025+0000 7fad5f7fe640 1 -- 172.21.15.60:0/2628339518 >> v1:172.21.15.60:6789/0 conn(0x7fad58054690 legacy=0x7fad580df5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.025+0000 7fad5dffb640 1 -- 172.21.15.60:0/2628339518 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 663865990 0 0) 0x7fad580e0090 con 0x7fad58054690 2024-05-09T05:00:35.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.025+0000 7fa3d27f0640 1 Processor -- start 2024-05-09T05:00:35.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.025+0000 7fa3d27f0640 1 -- start start 2024-05-09T05:00:35.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.025+0000 7fb406d9d640 1 --2- 172.21.15.60:0/2641843161 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4080ab290 0x7fb40816d150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb4080b4430 tx=0x7fb40002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf92d291823a8d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.026+0000 7f8872d78640 1 -- 172.21.15.60:0/209166590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886c15b520 msgr2=0x7f886c073790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.026+0000 7f8872d78640 1 --2- 172.21.15.60:0/209166590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886c15b520 0x7f886c073790 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f886400b1a0 tx=0x7f886402a980 comp rx=0 tx=0).stop 2024-05-09T05:00:35.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.026+0000 7f8872d78640 1 -- 172.21.15.60:0/209166590 shutdown_connections 2024-05-09T05:00:35.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.026+0000 7f8872d78640 1 --2- 172.21.15.60:0/209166590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886c15b520 0x7f886c073790 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.026+0000 7f8872d78640 1 --2- 172.21.15.60:0/209166590 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f886c0d0900 0x7f886c15c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.026+0000 7f8872d78640 1 -- 172.21.15.60:0/209166590 >> 172.21.15.60:0/209166590 conn(0x7f886c0b17d0 msgr2=0x7f886c06e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:35.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.026+0000 7fb40559a640 1 -- 172.21.15.60:0/2641843161 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb400007740 con 0x7fb4080ab290 2024-05-09T05:00:35.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.026+0000 7fb40559a640 1 -- 172.21.15.60:0/2641843161 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4000078e0 con 0x7fb4080ab290 2024-05-09T05:00:35.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.025+0000 7fad5f7fe640 1 -- 172.21.15.60:0/2628339518 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad580538d0 con 0x7fad580dfcb0 2024-05-09T05:00:35.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.026+0000 7fad5f7fe640 1 --2- 172.21.15.60:0/2628339518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad580dfcb0 0x7fad580e9e50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fad540097a0 tx=0x7fad5402b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b55b2e789525df8c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.027+0000 7fa3d27f0640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa3cc15b5d0 0x7fa3cc072a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.027+0000 7fad5dffb640 1 -- 172.21.15.60:0/2628339518 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad54038070 con 0x7fad580dfcb0 2024-05-09T05:00:35.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.027+0000 7fa3d27f0640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3cc06e7b0 0x7fa3cc072f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.027+0000 7fad5dffb640 1 -- 172.21.15.60:0/2628339518 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad54033750 con 0x7fad580dfcb0 2024-05-09T05:00:35.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.027+0000 7fb40759e640 1 -- 172.21.15.60:0/2641843161 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4080ab290 msgr2=0x7fb40816d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.027+0000 7fb40759e640 1 --2- 172.21.15.60:0/2641843161 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4080ab290 0x7fb40816d150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb4080b4430 tx=0x7fb40002a5f0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.027+0000 7fa3d0d8d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3cc06e7b0 0x7fa3cc072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7fa3d0d8d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3cc06e7b0 0x7fa3cc072f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.60:35050/0 (socket says 172.21.15.60:35050) 2024-05-09T05:00:35.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7fa3d0d8d640 1 -- 172.21.15.60:0/4020319482 learned_addr learned my addr 172.21.15.60:0/4020319482 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:35.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.027+0000 7eff8aff2640 1 Processor -- start 2024-05-09T05:00:35.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.027+0000 7eff8aff2640 1 -- start start 2024-05-09T05:00:35.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7fa3d27f0640 1 -- 172.21.15.60:0/4020319482 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3cc070a80 con 0x7fa3cc097de0 2024-05-09T05:00:35.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7fa3d27f0640 1 -- 172.21.15.60:0/4020319482 --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fa3cc06f9e0 con 0x7fa3cc15b5d0 2024-05-09T05:00:35.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7fb40759e640 1 -- 172.21.15.60:0/2641843161 shutdown_connections 2024-05-09T05:00:35.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7fb40759e640 1 --2- 172.21.15.60:0/2641843161 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4080ab290 0x7fb40816d150 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7fb40759e640 1 --2- 172.21.15.60:0/2641843161 >> v2:172.21.15.114:3300/0 conn(0x7fb40806fcb0 0x7fb408073ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7fb40759e640 1 -- 172.21.15.60:0/2641843161 >> 172.21.15.60:0/2641843161 conn(0x7fb4080b4000 msgr2=0x7fb40806e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:35.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7fad5dffb640 1 -- 172.21.15.60:0/2628339518 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad54033b20 con 0x7fad580dfcb0 2024-05-09T05:00:35.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7fa3d27f0640 1 -- 172.21.15.60:0/4020319482 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa3cc06fcc0 con 0x7fa3cc06e7b0 2024-05-09T05:00:35.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.026+0000 7f8872d78640 1 -- 172.21.15.60:0/209166590 shutdown_connections 2024-05-09T05:00:35.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.026+0000 7f8872d78640 1 -- 172.21.15.60:0/209166590 wait complete. 2024-05-09T05:00:35.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7fa3cbfff640 1 --2- 172.21.15.60:0/4020319482 >> v2:172.21.15.114:3300/0 conn(0x7fa3cc15b5d0 0x7fa3cc072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7f35b8e1d640 1 Processor -- start 2024-05-09T05:00:35.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7f8872d78640 1 Processor -- start 2024-05-09T05:00:35.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7f35b8e1d640 1 -- start start 2024-05-09T05:00:35.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7f2a5dbb5640 1 Processor -- start 2024-05-09T05:00:35.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7fad6530a640 1 -- 172.21.15.60:0/2628339518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad580dfcb0 msgr2=0x7fad580e9e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7fb40759e640 1 -- 172.21.15.60:0/2641843161 shutdown_connections 2024-05-09T05:00:35.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7fad6530a640 1 --2- 172.21.15.60:0/2628339518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad580dfcb0 0x7fad580e9e50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fad540097a0 tx=0x7fad5402b340 comp rx=0 tx=0).stop 2024-05-09T05:00:35.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7eff8aff2640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff8415b5b0 0x7eff8415c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7eff8aff2640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7eff8406c670 0x7eff840d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7eff8aff2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7eff840d0730 0x7eff8406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7eff8aff2640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7eff840d0b90 con 0x7eff840d0730 2024-05-09T05:00:35.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7eff8aff2640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7eff8415c950 con 0x7eff8406c670 2024-05-09T05:00:35.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7eff8aff2640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7eff840991f0 con 0x7eff8415b5b0 2024-05-09T05:00:35.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7f2a5dbb5640 1 -- start start 2024-05-09T05:00:35.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.029+0000 7eff89ff0640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff8415b5b0 0x7eff8415c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.029+0000 7eff897ef640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7eff8406c670 0x7eff840d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.029+0000 7eff8a7f1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7eff840d0730 0x7eff8406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.029+0000 7eff897ef640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7eff8406c670 0x7eff840d01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.60:41892/0 (socket says 172.21.15.60:41892) 2024-05-09T05:00:35.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.029+0000 7fad6530a640 1 -- 172.21.15.60:0/2628339518 shutdown_connections 2024-05-09T05:00:35.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.029+0000 7fad6530a640 1 --2- 172.21.15.60:0/2628339518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad580dfcb0 0x7fad580e9e50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.029+0000 7fad6530a640 1 -- 172.21.15.60:0/2628339518 >> 172.21.15.60:0/2628339518 conn(0x7fad58009fe0 msgr2=0x7fad5800a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:35.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.029+0000 7eff89ff0640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff8415b5b0 0x7eff8415c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.60:35064/0 (socket says 172.21.15.60:35064) 2024-05-09T05:00:35.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.029+0000 7eff89ff0640 1 -- 172.21.15.60:0/3673852414 learned_addr learned my addr 172.21.15.60:0/3673852414 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:35.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.029+0000 7eff8a7f1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7eff840d0730 0x7eff8406fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34986/0 (socket says 172.21.15.60:34986) 2024-05-09T05:00:35.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.029+0000 7fad6530a640 1 -- 172.21.15.60:0/2628339518 shutdown_connections 2024-05-09T05:00:35.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7f8872d78640 1 -- start start 2024-05-09T05:00:35.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.029+0000 7fad6530a640 1 -- 172.21.15.60:0/2628339518 wait complete. 2024-05-09T05:00:35.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.030+0000 7eff89ff0640 1 -- 172.21.15.60:0/3673852414 >> v2:172.21.15.114:3300/0 conn(0x7eff8406c670 msgr2=0x7eff840d01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:35.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.030+0000 7eff89ff0640 1 --2- 172.21.15.60:0/3673852414 >> v2:172.21.15.114:3300/0 conn(0x7eff8406c670 0x7eff840d01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.030+0000 7f2a5dbb5640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2a500069e0 0x7f2a50006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.030+0000 7f44bbf51640 1 Processor -- start 2024-05-09T05:00:35.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.030+0000 7eff89ff0640 1 -- 172.21.15.60:0/3673852414 >> v2:172.21.15.60:3300/0 conn(0x7eff840d0730 msgr2=0x7eff8406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:35.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.030+0000 7eff89ff0640 1 --2- 172.21.15.60:0/3673852414 >> v2:172.21.15.60:3300/0 conn(0x7eff840d0730 0x7eff8406fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.030+0000 7f2a5dbb5640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a50053720 con 0x7f2a50005e90 2024-05-09T05:00:35.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.030+0000 7f2a5dbb5640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a50053a00 con 0x7f2a500ead00 2024-05-09T05:00:35.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.030+0000 7f2a5dbb5640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f2a50054330 con 0x7f2a500069e0 2024-05-09T05:00:35.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.030+0000 7f44bbf51640 1 -- start start 2024-05-09T05:00:35.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.030+0000 7eff89ff0640 1 -- 172.21.15.60:0/3673852414 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff840985e0 con 0x7eff8415b5b0 2024-05-09T05:00:35.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.030+0000 7f2a5cbb3640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2a500ead00 0x7f2a5004fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.60:45302/0 (socket says 172.21.15.60:45302) 2024-05-09T05:00:35.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.030+0000 7f2a5cbb3640 1 -- 172.21.15.60:0/2771770671 learned_addr learned my addr 172.21.15.60:0/2771770671 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:35.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7fb40759e640 1 -- 172.21.15.60:0/2641843161 wait complete. 2024-05-09T05:00:35.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.030+0000 7eff8a7f1640 1 --2- 172.21.15.60:0/3673852414 >> v2:172.21.15.60:3300/0 conn(0x7eff840d0730 0x7eff8406fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:00:35.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7fa3caffd640 1 -- 172.21.15.60:0/4020319482 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3343958319 0 0) 0x7fa3cc070a80 con 0x7fa3cc097de0 2024-05-09T05:00:35.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.030+0000 7f35b8e1d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f35b40ad040 0x7f35b40b2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.028+0000 7fa3caffd640 1 -- 172.21.15.60:0/4020319482 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3b4003520 con 0x7fa3cc097de0 2024-05-09T05:00:35.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.029+0000 7fa3caffd640 1 -- 172.21.15.60:0/4020319482 <== mon.1 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2918093628 0 0) 0x7fa3b4003520 con 0x7fa3cc097de0 2024-05-09T05:00:35.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.029+0000 7fa3d0d8d640 1 -- 172.21.15.60:0/4020319482 >> v2:172.21.15.114:3300/0 conn(0x7fa3cc15b5d0 msgr2=0x7fa3cc072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:35.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.029+0000 7fa3d0d8d640 1 --2- 172.21.15.60:0/4020319482 >> v2:172.21.15.114:3300/0 conn(0x7fa3cc15b5d0 0x7fa3cc072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.029+0000 7fa3d0d8d640 1 -- 172.21.15.60:0/4020319482 >> v1:172.21.15.114:6789/0 conn(0x7fa3cc097de0 legacy=0x7fa3cc071810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.029+0000 7fa3d0d8d640 1 -- 172.21.15.60:0/4020319482 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3cc06ffa0 con 0x7fa3cc06e7b0 2024-05-09T05:00:35.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.030+0000 7fb40759e640 1 Processor -- start 2024-05-09T05:00:35.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f8872d78640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886c06c670 0x7f886c0ae970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f8872d78640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f886c0d0900 0x7f886c14f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f8872d78640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f886c15b520 0x7f886c0a9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f8872d78640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f886c098270 con 0x7f886c06c670 2024-05-09T05:00:35.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f8872d78640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f886c098e80 con 0x7f886c15b520 2024-05-09T05:00:35.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f8872d78640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f886c0719c0 con 0x7f886c0d0900 2024-05-09T05:00:35.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f44bbf51640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f44b40aecf0 0x7f44b40b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f44bbf51640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44b40b0980 0x7f44b406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7fad6530a640 1 Processor -- start 2024-05-09T05:00:35.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.030+0000 7f2a577fe640 1 -- 172.21.15.60:0/2771770671 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1715507082 0 0) 0x7f2a50053a00 con 0x7f2a500ead00 2024-05-09T05:00:35.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f2a577fe640 1 -- 172.21.15.60:0/2771770671 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a48003520 con 0x7f2a500ead00 2024-05-09T05:00:35.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f2a577fe640 1 -- 172.21.15.60:0/2771770671 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 491941177 0 0) 0x7f2a50053720 con 0x7f2a50005e90 2024-05-09T05:00:35.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f2a577fe640 1 -- 172.21.15.60:0/2771770671 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a50053a00 con 0x7f2a50005e90 2024-05-09T05:00:35.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f2a577fe640 1 -- 172.21.15.60:0/2771770671 <== mon.1 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2075204927 0 0) 0x7f2a48003520 con 0x7f2a500ead00 2024-05-09T05:00:35.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f2a577fe640 1 -- 172.21.15.60:0/2771770671 >> v2:172.21.15.114:3300/0 conn(0x7f2a500069e0 msgr2=0x7f2a50006e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:00:35.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f2a577fe640 1 --2- 172.21.15.60:0/2771770671 >> v2:172.21.15.114:3300/0 conn(0x7f2a500069e0 0x7f2a50006e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f2a577fe640 1 -- 172.21.15.60:0/2771770671 >> v1:172.21.15.60:6789/0 conn(0x7f2a50005e90 legacy=0x7f2a500062d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7fad6530a640 1 -- start start 2024-05-09T05:00:35.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7eff89ff0640 1 --2- 172.21.15.60:0/3673852414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff8415b5b0 0x7eff8415c410 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7eff8406c580 tx=0x7eff7c02a6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1468354ab9651f44 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7eff88fee640 1 -- 172.21.15.60:0/3673852414 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7c036070 con 0x7eff8415b5b0 2024-05-09T05:00:35.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7eff88fee640 1 -- 172.21.15.60:0/3673852414 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff7c031600 con 0x7eff8415b5b0 2024-05-09T05:00:35.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7eff88fee640 1 -- 172.21.15.60:0/3673852414 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7c0319b0 con 0x7eff8415b5b0 2024-05-09T05:00:35.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7eff8aff2640 1 -- 172.21.15.60:0/3673852414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff8415b5b0 msgr2=0x7eff8415c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7eff8aff2640 1 --2- 172.21.15.60:0/3673852414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff8415b5b0 0x7eff8415c410 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7eff8406c580 tx=0x7eff7c02a6b0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7eff8aff2640 1 -- 172.21.15.60:0/3673852414 shutdown_connections 2024-05-09T05:00:35.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7eff8aff2640 1 --2- 172.21.15.60:0/3673852414 >> v2:172.21.15.114:3300/0 conn(0x7eff8406c670 0x7eff840d01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7eff8aff2640 1 --2- 172.21.15.60:0/3673852414 >> v2:172.21.15.60:3300/0 conn(0x7eff840d0730 0x7eff8406fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7eff8aff2640 1 --2- 172.21.15.60:0/3673852414 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff8415b5b0 0x7eff8415c410 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7eff8aff2640 1 -- 172.21.15.60:0/3673852414 >> 172.21.15.60:0/3673852414 conn(0x7eff840b17d0 msgr2=0x7eff8406e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:35.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7eff8aff2640 1 -- 172.21.15.60:0/3673852414 shutdown_connections 2024-05-09T05:00:35.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7eff8aff2640 1 -- 172.21.15.60:0/3673852414 wait complete. 2024-05-09T05:00:35.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7fa3d0d8d640 1 --2- 172.21.15.60:0/4020319482 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3cc06e7b0 0x7fa3cc072f70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400b230 tx=0x7fa3c402aac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e98101788320482 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7fa3caffd640 1 -- 172.21.15.60:0/4020319482 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4033020 con 0x7fa3cc06e7b0 2024-05-09T05:00:35.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7fa3caffd640 1 -- 172.21.15.60:0/4020319482 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3c402f660 con 0x7fa3cc06e7b0 2024-05-09T05:00:35.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7fa3caffd640 1 -- 172.21.15.60:0/4020319482 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c00016f0 con 0x7fa3cc06e7b0 2024-05-09T05:00:35.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7fa3d27f0640 1 -- 172.21.15.60:0/4020319482 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3cc06e7b0 msgr2=0x7fa3cc072f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7fa3d27f0640 1 --2- 172.21.15.60:0/4020319482 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3cc06e7b0 0x7fa3cc072f70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400b230 tx=0x7fa3c402aac0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fa3d27f0640 1 -- 172.21.15.60:0/4020319482 shutdown_connections 2024-05-09T05:00:35.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fa3d27f0640 1 --2- 172.21.15.60:0/4020319482 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3cc06e7b0 0x7fa3cc072f70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fa3d27f0640 1 --2- 172.21.15.60:0/4020319482 >> v2:172.21.15.114:3300/0 conn(0x7fa3cc15b5d0 0x7fa3cc072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fa3d27f0640 1 -- 172.21.15.60:0/4020319482 >> 172.21.15.60:0/4020319482 conn(0x7fa3cc0972f0 msgr2=0x7fa3cc0976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:35.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fa3d27f0640 1 -- 172.21.15.60:0/4020319482 shutdown_connections 2024-05-09T05:00:35.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7fa3d27f0640 1 -- 172.21.15.60:0/4020319482 wait complete. 2024-05-09T05:00:35.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f2a577fe640 1 -- 172.21.15.60:0/2771770671 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a50053ce0 con 0x7f2a500ead00 2024-05-09T05:00:35.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f2a577fe640 1 -- 172.21.15.60:0/2771770671 <== mon.1 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f2a4c0016f0 con 0x7f2a500ead00 2024-05-09T05:00:35.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f2a577fe640 1 -- 172.21.15.60:0/2771770671 >> v1:172.21.15.114:6789/0 conn(0x7f2a500ead00 legacy=0x7f2a5004fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f2a577fe640 1 --2- 172.21.15.60:0/2771770671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a48003c30 0x7f2a48004070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f2a577fe640 1 --2- 172.21.15.60:0/2771770671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a48004800 0x7f2a48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f2a577fe640 1 --2- 172.21.15.60:0/2771770671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2a480053c0 0x7f2a4801c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f2a577fe640 1 -- 172.21.15.60:0/2771770671 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2a50054330 con 0x7f2a48004800 2024-05-09T05:00:35.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f2a577fe640 1 -- 172.21.15.60:0/2771770671 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4801d290 con 0x7f2a48003c30 2024-05-09T05:00:35.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f2a577fe640 1 -- 172.21.15.60:0/2771770671 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f2a4801d490 con 0x7f2a480053c0 2024-05-09T05:00:35.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f2a5cbb3640 1 --2- 172.21.15.60:0/2771770671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a48003c30 0x7f2a48004070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f2a5d3b4640 1 --2- 172.21.15.60:0/2771770671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2a480053c0 0x7f2a4801c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f2a5cbb3640 1 -- 172.21.15.60:0/2771770671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2a480053c0 msgr2=0x7f2a4801c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:35.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f2a5cbb3640 1 --2- 172.21.15.60:0/2771770671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2a480053c0 0x7f2a4801c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f2a5cbb3640 1 -- 172.21.15.60:0/2771770671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a48004800 msgr2=0x7f2a48004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:00:35.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7f2a57fff640 1 --2- 172.21.15.60:0/2771770671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a48004800 0x7f2a48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7f2a5cbb3640 1 --2- 172.21.15.60:0/2771770671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a48004800 0x7f2a48004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7f2a5cbb3640 1 -- 172.21.15.60:0/2771770671 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a50053ce0 con 0x7f2a48003c30 2024-05-09T05:00:35.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7f2a57fff640 1 --2- 172.21.15.60:0/2771770671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a48004800 0x7f2a48004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:35.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7f2a5cbb3640 1 --2- 172.21.15.60:0/2771770671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a48003c30 0x7f2a48004070 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c00c1c0 tx=0x7f2a48005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86ea2a475354e374 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7f2a5d3b4640 1 -- 172.21.15.60:0/2771770671 reap_dead start 2024-05-09T05:00:35.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7f2a577fe640 1 -- 172.21.15.60:0/2771770671 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c007d00 con 0x7f2a48003c30 2024-05-09T05:00:35.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7f2a577fe640 1 -- 172.21.15.60:0/2771770671 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a4c005af0 con 0x7f2a48003c30 2024-05-09T05:00:35.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7f2a577fe640 1 -- 172.21.15.60:0/2771770671 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c007070 con 0x7f2a48003c30 2024-05-09T05:00:35.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.035+0000 7f2a5dbb5640 1 -- 172.21.15.60:0/2771770671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a48003c30 msgr2=0x7f2a48004070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f44bbf51640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44b40ad930 con 0x7f44b41637d0 2024-05-09T05:00:35.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f44bbf51640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f44b4163bb0 con 0x7f44b40aecf0 2024-05-09T05:00:35.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f44bbf51640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f44b4072130 con 0x7f44b40b0980 2024-05-09T05:00:35.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f44b9ced640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f44b41637d0 0x7f44b40728a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38492/0 (socket says 172.21.15.60:38492) 2024-05-09T05:00:35.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f44b9ced640 1 -- 172.21.15.60:0/3167056078 learned_addr learned my addr 172.21.15.60:0/3167056078 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:35.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f44b94ec640 1 --2- 172.21.15.60:0/3167056078 >> v2:172.21.15.60:3300/0 conn(0x7f44b40aecf0 0x7f44b40b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f44ba4ee640 1 --2- 172.21.15.60:0/3167056078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44b40b0980 0x7f44b406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f44b94ec640 1 -- 172.21.15.60:0/3167056078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44b40b0980 msgr2=0x7f44b406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:35.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f44b94ec640 1 --2- 172.21.15.60:0/3167056078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44b40b0980 0x7f44b406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f44b94ec640 1 -- 172.21.15.60:0/3167056078 >> v1:172.21.15.60:6789/0 conn(0x7f44b41637d0 legacy=0x7f44b40728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f44b94ec640 1 -- 172.21.15.60:0/3167056078 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b40adc10 con 0x7f44b40aecf0 2024-05-09T05:00:35.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7f44b94ec640 1 --2- 172.21.15.60:0/3167056078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b40aecf0 0x7f44b40b0440 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f44a40097a0 tx=0x7f44a403b190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=666dd1c8ce19eec4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.035+0000 7f44b8ceb640 1 -- 172.21.15.60:0/3167056078 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a4048070 con 0x7f44b40aecf0 2024-05-09T05:00:35.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.035+0000 7f44b8ceb640 1 -- 172.21.15.60:0/3167056078 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44a4043440 con 0x7f44b40aecf0 2024-05-09T05:00:35.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.035+0000 7f44b8ceb640 1 -- 172.21.15.60:0/3167056078 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a4043810 con 0x7f44b40aecf0 2024-05-09T05:00:35.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7f44bbf51640 1 -- 172.21.15.60:0/3167056078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b40aecf0 msgr2=0x7f44b40b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7f44bbf51640 1 --2- 172.21.15.60:0/3167056078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b40aecf0 0x7f44b40b0440 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f44a40097a0 tx=0x7f44a403b190 comp rx=0 tx=0).stop 2024-05-09T05:00:35.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7f44bbf51640 1 -- 172.21.15.60:0/3167056078 shutdown_connections 2024-05-09T05:00:35.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7f44bbf51640 1 --2- 172.21.15.60:0/3167056078 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44b40b0980 0x7f44b406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7f44bbf51640 1 --2- 172.21.15.60:0/3167056078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b40aecf0 0x7f44b40b0440 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.035+0000 7f2a5dbb5640 1 --2- 172.21.15.60:0/2771770671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a48003c30 0x7f2a48004070 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c00c1c0 tx=0x7f2a48005830 comp rx=0 tx=0).stop 2024-05-09T05:00:35.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7f2a5dbb5640 1 -- 172.21.15.60:0/2771770671 shutdown_connections 2024-05-09T05:00:35.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7f2a5dbb5640 1 --2- 172.21.15.60:0/2771770671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a48003c30 0x7f2a48004070 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7f2a5dbb5640 1 -- 172.21.15.60:0/2771770671 >> 172.21.15.60:0/2771770671 conn(0x7f2a50009fb0 msgr2=0x7f2a5000a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:35.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7f2a5dbb5640 1 -- 172.21.15.60:0/2771770671 shutdown_connections 2024-05-09T05:00:35.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7f2a5dbb5640 1 -- 172.21.15.60:0/2771770671 wait complete. 2024-05-09T05:00:35.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7f44bbf51640 1 -- 172.21.15.60:0/3167056078 >> 172.21.15.60:0/3167056078 conn(0x7f44b4073240 msgr2=0x7f44b4073650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:35.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7f44bbf51640 1 -- 172.21.15.60:0/3167056078 shutdown_connections 2024-05-09T05:00:35.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.037+0000 7f44bbf51640 1 -- 172.21.15.60:0/3167056078 wait complete. 2024-05-09T05:00:35.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f8871575640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f886c0d0900 0x7f886c14f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f8871575640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f886c0d0900 0x7f886c14f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.60:35076/0 (socket says 172.21.15.60:35076) 2024-05-09T05:00:35.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f8871575640 1 -- 172.21.15.60:0/2256816143 learned_addr learned my addr 172.21.15.60:0/2256816143 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:35.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f8871d76640 1 --2- 172.21.15.60:0/2256816143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886c06c670 0x7f886c0ae970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f8871575640 1 -- 172.21.15.60:0/2256816143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f886c15b520 msgr2=0x7f886c0a9000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:00:35.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f8871575640 1 --2- 172.21.15.60:0/2256816143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f886c15b520 0x7f886c0a9000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f8871575640 1 -- 172.21.15.60:0/2256816143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886c06c670 msgr2=0x7f886c0ae970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f8871575640 1 --2- 172.21.15.60:0/2256816143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886c06c670 0x7f886c0ae970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f8871575640 1 -- 172.21.15.60:0/2256816143 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886c0aefe0 con 0x7f886c0d0900 2024-05-09T05:00:35.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f8871575640 1 --2- 172.21.15.60:0/2256816143 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f886c0d0900 0x7f886c14f910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f886800b450 tx=0x7f886800b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7f88627fc640 1 -- 172.21.15.60:0/2256816143 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8868015020 con 0x7f886c0d0900 2024-05-09T05:00:35.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7f8872d78640 1 -- 172.21.15.60:0/2256816143 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f886c0af2c0 con 0x7f886c0d0900 2024-05-09T05:00:35.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7f8872d78640 1 -- 172.21.15.60:0/2256816143 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f886c0a9540 con 0x7f886c0d0900 2024-05-09T05:00:35.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7f88627fc640 1 -- 172.21.15.60:0/2256816143 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88680118b0 con 0x7f886c0d0900 2024-05-09T05:00:35.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7f88627fc640 1 -- 172.21.15.60:0/2256816143 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8868011ca0 con 0x7f886c0d0900 2024-05-09T05:00:35.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7f8872d78640 1 -- 172.21.15.60:0/2256816143 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8854005290 con 0x7f886c0d0900 2024-05-09T05:00:35.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.037+0000 7f88627fc640 1 -- 172.21.15.60:0/2256816143 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8868018020 con 0x7f886c0d0900 2024-05-09T05:00:35.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.037+0000 7f88627fc640 1 --2- 172.21.15.60:0/2256816143 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8850076350 0x7f8850078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.037+0000 7f88627fc640 1 -- 172.21.15.60:0/2256816143 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f8868097820 con 0x7f886c0d0900 2024-05-09T05:00:35.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.037+0000 7f8871d76640 1 --2- 172.21.15.60:0/2256816143 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8850076350 0x7f8850078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.038+0000 7f8871d76640 1 --2- 172.21.15.60:0/2256816143 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8850076350 0x7f8850078810 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f885c006fd0 tx=0x7f885c009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.042+0000 7f88627fc640 1 -- 172.21.15.60:0/2256816143 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f886801f020 con 0x7f886c0d0900 2024-05-09T05:00:35.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7fb40759e640 1 -- start start 2024-05-09T05:00:35.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7fb40759e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb40806fcb0 0x7fb4080cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7fb40759e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb408073040 0x7fb4080ae260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7fb40759e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4080ae7a0 0x7fb408161980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7fb40759e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb4080707a0 con 0x7fb4080ae7a0 2024-05-09T05:00:35.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7fb406d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4080ae7a0 0x7fb408161980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7fb40759e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb408074400 con 0x7fb408073040 2024-05-09T05:00:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7fb40759e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb408070920 con 0x7fb40806fcb0 2024-05-09T05:00:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7fb405d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb408073040 0x7fb4080ae260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7fb406d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4080ae7a0 0x7fb408161980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35034/0 (socket says 172.21.15.60:35034) 2024-05-09T05:00:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7fb406d9d640 1 -- 172.21.15.60:0/1121137233 learned_addr learned my addr 172.21.15.60:0/1121137233 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fb40659c640 1 --2- 172.21.15.60:0/1121137233 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb40806fcb0 0x7fb4080cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fb406d9d640 1 -- 172.21.15.60:0/1121137233 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb40806fcb0 msgr2=0x7fb4080cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fb406d9d640 1 --2- 172.21.15.60:0/1121137233 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb40806fcb0 0x7fb4080cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fb406d9d640 1 -- 172.21.15.60:0/1121137233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb408073040 msgr2=0x7fb4080ae260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fb406d9d640 1 --2- 172.21.15.60:0/1121137233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb408073040 0x7fb4080ae260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fb406d9d640 1 -- 172.21.15.60:0/1121137233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4080ad1c0 con 0x7fb4080ae7a0 2024-05-09T05:00:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fb405d9b640 1 --2- 172.21.15.60:0/1121137233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb408073040 0x7fb4080ae260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:00:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fb406d9d640 1 --2- 172.21.15.60:0/1121137233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4080ae7a0 0x7fb408161980 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb40002abe0 tx=0x7fb400004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fb3f6ffd640 1 -- 172.21.15.60:0/1121137233 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb400061070 con 0x7fb4080ae7a0 2024-05-09T05:00:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fb3f6ffd640 1 -- 172.21.15.60:0/1121137233 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb400031e40 con 0x7fb4080ae7a0 2024-05-09T05:00:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fb3f6ffd640 1 -- 172.21.15.60:0/1121137233 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4000666a0 con 0x7fb4080ae7a0 2024-05-09T05:00:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7fb40759e640 1 -- 172.21.15.60:0/1121137233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4080ad4a0 con 0x7fb4080ae7a0 2024-05-09T05:00:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7fb40759e640 1 -- 172.21.15.60:0/1121137233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4080ada60 con 0x7fb4080ae7a0 2024-05-09T05:00:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7fb40759e640 1 -- 172.21.15.60:0/1121137233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4080640c0 con 0x7fb4080ae7a0 2024-05-09T05:00:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7fb3f6ffd640 1 -- 172.21.15.60:0/1121137233 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb400066840 con 0x7fb4080ae7a0 2024-05-09T05:00:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.037+0000 7fb3f6ffd640 1 --2- 172.21.15.60:0/1121137233 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb3e8076350 0x7fb3e8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.037+0000 7fb3f6ffd640 1 -- 172.21.15.60:0/1121137233 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb4000db060 con 0x7fb4080ae7a0 2024-05-09T05:00:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.037+0000 7fb40659c640 1 --2- 172.21.15.60:0/1121137233 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb3e8076350 0x7fb3e8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.038+0000 7fb40659c640 1 --2- 172.21.15.60:0/1121137233 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb3e8076350 0x7fb3e8078810 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc009950 tx=0x7fb3fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.042+0000 7fb3f6ffd640 1 -- 172.21.15.60:0/1121137233 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb400009a20 con 0x7fb4080ae7a0 2024-05-09T05:00:35.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f35b8e1d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f35b40688d0 0x7f35b4141400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f35b8e1d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35b415fd10 con 0x7f35b40b2670 2024-05-09T05:00:35.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f35b8e1d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f35b40b2ad0 con 0x7f35b40688d0 2024-05-09T05:00:35.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.031+0000 7f35b8e1d640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f35b4141a60 con 0x7f35b40ad040 2024-05-09T05:00:35.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f35b2d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f35b40688d0 0x7f35b4141400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f35b2d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f35b40688d0 0x7f35b4141400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34998/0 (socket says 172.21.15.60:34998) 2024-05-09T05:00:35.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f35b2d9d640 1 -- 172.21.15.60:0/2280379358 learned_addr learned my addr 172.21.15.60:0/2280379358 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:35.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f35b259c640 1 --2- 172.21.15.60:0/2280379358 >> v2:172.21.15.114:3300/0 conn(0x7f35b40ad040 0x7f35b40b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f35b259c640 1 -- 172.21.15.60:0/2280379358 >> v2:172.21.15.60:3300/0 conn(0x7f35b40688d0 msgr2=0x7f35b4141400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:35.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f35b259c640 1 --2- 172.21.15.60:0/2280379358 >> v2:172.21.15.60:3300/0 conn(0x7f35b40688d0 0x7f35b4141400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f35b259c640 1 -- 172.21.15.60:0/2280379358 >> v1:172.21.15.60:6789/0 conn(0x7f35b40b2670 legacy=0x7f35b4140cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.032+0000 7f35b259c640 1 -- 172.21.15.60:0/2280379358 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b415ef50 con 0x7f35b40ad040 2024-05-09T05:00:35.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7f35b259c640 1 --2- 172.21.15.60:0/2280379358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35b40ad040 0x7f35b40b2130 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f35b40b4370 tx=0x7f35ac03b620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff93fed69c338772 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7f35b159a640 1 -- 172.21.15.60:0/2280379358 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac007740 con 0x7f35b40ad040 2024-05-09T05:00:35.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7f35b159a640 1 -- 172.21.15.60:0/2280379358 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35ac0078e0 con 0x7f35b40ad040 2024-05-09T05:00:35.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.035+0000 7f35b159a640 1 -- 172.21.15.60:0/2280379358 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac007cb0 con 0x7f35b40ad040 2024-05-09T05:00:35.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.035+0000 7f35b8e1d640 1 -- 172.21.15.60:0/2280379358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35b40ad040 msgr2=0x7f35b40b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.035+0000 7f35b8e1d640 1 --2- 172.21.15.60:0/2280379358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35b40ad040 0x7f35b40b2130 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f35b40b4370 tx=0x7f35ac03b620 comp rx=0 tx=0).stop 2024-05-09T05:00:35.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7f35b8e1d640 1 -- 172.21.15.60:0/2280379358 shutdown_connections 2024-05-09T05:00:35.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7f35b8e1d640 1 --2- 172.21.15.60:0/2280379358 >> v2:172.21.15.60:3300/0 conn(0x7f35b40688d0 0x7f35b4141400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7f35b8e1d640 1 --2- 172.21.15.60:0/2280379358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35b40ad040 0x7f35b40b2130 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7f35b8e1d640 1 -- 172.21.15.60:0/2280379358 >> 172.21.15.60:0/2280379358 conn(0x7f35b40684c0 msgr2=0x7f35b406fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:35.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7f35b8e1d640 1 -- 172.21.15.60:0/2280379358 shutdown_connections 2024-05-09T05:00:35.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7f35b8e1d640 1 -- 172.21.15.60:0/2280379358 wait complete. 2024-05-09T05:00:35.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fad6530a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad58054690 0x7fad5804e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fad6530a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad580df1c0 0x7fad58185420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fad5effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad58054690 0x7fad5804e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.033+0000 7fad5effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad58054690 0x7fad5804e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35044/0 (socket says 172.21.15.60:35044) 2024-05-09T05:00:35.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7fad6530a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad580dfcb0 0x7fad5802b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7fad5e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad580df1c0 0x7fad58185420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7fad5e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad580df1c0 0x7fad58185420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.60:41932/0 (socket says 172.21.15.60:41932) 2024-05-09T05:00:35.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7fad5e7fc640 1 -- 172.21.15.60:0/2152312270 learned_addr learned my addr 172.21.15.60:0/2152312270 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:35.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7fad6530a640 1 -- 172.21.15.60:0/2152312270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fad58054170 con 0x7fad58054690 2024-05-09T05:00:35.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7fad6530a640 1 -- 172.21.15.60:0/2152312270 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fad58053e90 con 0x7fad580df1c0 2024-05-09T05:00:35.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7fad6530a640 1 -- 172.21.15.60:0/2152312270 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fad580e04e0 con 0x7fad580dfcb0 2024-05-09T05:00:35.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7fad5f7fe640 1 --2- 172.21.15.60:0/2152312270 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad580dfcb0 0x7fad5802b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7fad5e7fc640 1 -- 172.21.15.60:0/2152312270 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad580dfcb0 msgr2=0x7fad5802b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7fad5e7fc640 1 --2- 172.21.15.60:0/2152312270 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad580dfcb0 0x7fad5802b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7fad5e7fc640 1 -- 172.21.15.60:0/2152312270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad58054690 msgr2=0x7fad5804e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.066+0000 7eff8aff2640 1 Processor -- start 2024-05-09T05:00:35.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7fad5e7fc640 1 --2- 172.21.15.60:0/2152312270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad58054690 0x7fad5804e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7fad5e7fc640 1 -- 172.21.15.60:0/2152312270 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad5804ea40 con 0x7fad580df1c0 2024-05-09T05:00:35.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.034+0000 7fad5e7fc640 1 --2- 172.21.15.60:0/2152312270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad580df1c0 0x7fad58185420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fad60050ac0 tx=0x7fad60051220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.035+0000 7fad477fe640 1 -- 172.21.15.60:0/2152312270 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad600ba070 con 0x7fad580df1c0 2024-05-09T05:00:35.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.066+0000 7f2a5dbb5640 1 Processor -- start 2024-05-09T05:00:35.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.035+0000 7fad477fe640 1 -- 172.21.15.60:0/2152312270 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad60075e60 con 0x7fad580df1c0 2024-05-09T05:00:35.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.035+0000 7fad477fe640 1 -- 172.21.15.60:0/2152312270 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad600713a0 con 0x7fad580df1c0 2024-05-09T05:00:35.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.035+0000 7fad6530a640 1 -- 172.21.15.60:0/2152312270 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad580490b0 con 0x7fad580df1c0 2024-05-09T05:00:35.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.036+0000 7fad6530a640 1 -- 172.21.15.60:0/2152312270 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad58049670 con 0x7fad580df1c0 2024-05-09T05:00:35.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.037+0000 7fad6530a640 1 -- 172.21.15.60:0/2152312270 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad58008a00 con 0x7fad580df1c0 2024-05-09T05:00:35.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.067+0000 7fa3d27f0640 1 Processor -- start 2024-05-09T05:00:35.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.037+0000 7fad477fe640 1 -- 172.21.15.60:0/2152312270 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fad60075860 con 0x7fad580df1c0 2024-05-09T05:00:35.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.038+0000 7fad477fe640 1 --2- 172.21.15.60:0/2152312270 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fad4c0763a0 0x7fad4c078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.038+0000 7fad477fe640 1 -- 172.21.15.60:0/2152312270 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fad60113130 con 0x7fad580df1c0 2024-05-09T05:00:35.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.038+0000 7fad5effd640 1 --2- 172.21.15.60:0/2152312270 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fad4c0763a0 0x7fad4c078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.039+0000 7fad5effd640 1 --2- 172.21.15.60:0/2152312270 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fad4c0763a0 0x7fad4c078860 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fad50000f60 tx=0x7fad50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.043+0000 7fad477fe640 1 -- 172.21.15.60:0/2152312270 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fad600dc170 con 0x7fad580df1c0 2024-05-09T05:00:35.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.067+0000 7fa3d27f0640 1 -- start start 2024-05-09T05:00:35.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.067+0000 7f44bbf51640 1 Processor -- start 2024-05-09T05:00:35.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.067+0000 7f35b8e1d640 1 Processor -- start 2024-05-09T05:00:35.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.066+0000 7eff8aff2640 1 -- start start 2024-05-09T05:00:35.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.067+0000 7f44bbf51640 1 -- start start 2024-05-09T05:00:35.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.068+0000 7eff8aff2640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff8406c670 0x7eff840af000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.068+0000 7eff8aff2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff840d0730 0x7eff84152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.067+0000 7f2a5dbb5640 1 -- start start 2024-05-09T05:00:35.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.068+0000 7f2a5dbb5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a48003c30 0x7f2a500dad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.068+0000 7f2a5dbb5640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2a5004bb90 0x7f2a5004c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.068+0000 7f2a5dbb5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a500d1fa0 0x7f2a500d2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.068+0000 7f2a5dbb5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2a50054940 con 0x7f2a500d1fa0 2024-05-09T05:00:35.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.068+0000 7f2a5dbb5640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2a500547c0 con 0x7f2a48003c30 2024-05-09T05:00:35.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.068+0000 7f2a5dbb5640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f2a5004f550 con 0x7f2a5004bb90 2024-05-09T05:00:35.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7f2a5d3b4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a500d1fa0 0x7f2a500d2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7f2a5d3b4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a500d1fa0 0x7f2a500d2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35054/0 (socket says 172.21.15.60:35054) 2024-05-09T05:00:35.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7f2a5d3b4640 1 -- 172.21.15.60:0/4009042441 learned_addr learned my addr 172.21.15.60:0/4009042441 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:35.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.068+0000 7eff89ff0640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff8406c670 0x7eff840af000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7fa3d27f0640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3cc06e7b0 0x7fa3cc0cf790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7f2a5cbb3640 1 --2- 172.21.15.60:0/4009042441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a48003c30 0x7f2a500dad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7f44bbf51640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44b40aecf0 0x7f44b414e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7f44bbf51640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44b40b0980 0x7f44b4164700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7f44bbf51640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b41637d0 0x7f44b406fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.068+0000 7eff89ff0640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff8406c670 0x7eff840af000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.60:35128/0 (socket says 172.21.15.60:35128) 2024-05-09T05:00:35.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.068+0000 7eff89ff0640 1 -- 172.21.15.60:0/1526119212 learned_addr learned my addr 172.21.15.60:0/1526119212 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:35.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7eff897ef640 1 --2- 172.21.15.60:0/1526119212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff840d0730 0x7eff84152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7eff8aff2640 1 --2- 172.21.15.60:0/1526119212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff8415b5b0 0x7eff840a75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7eff8aff2640 1 -- 172.21.15.60:0/1526119212 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7eff84098e80 con 0x7eff840d0730 2024-05-09T05:00:35.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7eff8aff2640 1 -- 172.21.15.60:0/1526119212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7eff84098ba0 con 0x7eff8415b5b0 2024-05-09T05:00:35.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7eff8aff2640 1 -- 172.21.15.60:0/1526119212 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7eff84098270 con 0x7eff8406c670 2024-05-09T05:00:35.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7eff8a7f1640 1 --2- 172.21.15.60:0/1526119212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff8415b5b0 0x7eff840a75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7fa3d27f0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3cc097de0 0x7fa3cc0ac8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7fa3cbfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3cc06e7b0 0x7fa3cc0cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7fa3cbfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3cc06e7b0 0x7fa3cc0cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.60:35144/0 (socket says 172.21.15.60:35144) 2024-05-09T05:00:35.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7fa3cbfff640 1 -- 172.21.15.60:0/503245253 learned_addr learned my addr 172.21.15.60:0/503245253 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:35.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7fa3d27f0640 1 --2- 172.21.15.60:0/503245253 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3cc15b5d0 0x7fa3cc0ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.068+0000 7f35b8e1d640 1 -- start start 2024-05-09T05:00:35.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7f35b8e1d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35b40688d0 0x7f35b406dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7f35b8e1d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35b40ad040 0x7f35b406e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7f35b8e1d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35b40b2670 0x7f35b41550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7f35b8e1d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f35b415f7f0 con 0x7f35b40ad040 2024-05-09T05:00:35.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7f35b8e1d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f35b415f510 con 0x7f35b40688d0 2024-05-09T05:00:35.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7f35b8e1d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f35b40ad420 con 0x7f35b40b2670 2024-05-09T05:00:35.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7f35b2d9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35b40b2670 0x7f35b41550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f44bbf51640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f44b406dd60 con 0x7f44b41637d0 2024-05-09T05:00:35.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f44b94ec640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44b40b0980 0x7f44b4164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f44b94ec640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44b40b0980 0x7f44b4164700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.60:41988/0 (socket says 172.21.15.60:41988) 2024-05-09T05:00:35.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f44b94ec640 1 -- 172.21.15.60:0/1917377462 learned_addr learned my addr 172.21.15.60:0/1917377462 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:35.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f44bbf51640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f44b4072380 con 0x7f44b40b0980 2024-05-09T05:00:35.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f44bbf51640 1 -- 172.21.15.60:0/1917377462 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f44b406dee0 con 0x7f44b40aecf0 2024-05-09T05:00:35.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7fa3d27f0640 1 -- 172.21.15.60:0/503245253 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa3cc06f6a0 con 0x7fa3cc097de0 2024-05-09T05:00:35.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7fa3d0d8d640 1 --2- 172.21.15.60:0/503245253 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3cc15b5d0 0x7fa3cc0ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f2a5d3b4640 1 -- 172.21.15.60:0/4009042441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2a5004bb90 msgr2=0x7f2a5004c010 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T05:00:35.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7fa3d27f0640 1 -- 172.21.15.60:0/503245253 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa3cc070560 con 0x7fa3cc15b5d0 2024-05-09T05:00:35.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f2a5d3b4640 1 --2- 172.21.15.60:0/4009042441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2a5004bb90 0x7f2a5004c010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f2a5d3b4640 1 -- 172.21.15.60:0/4009042441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a48003c30 msgr2=0x7f2a500dad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7fa3cb7fe640 1 --2- 172.21.15.60:0/503245253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3cc097de0 0x7fa3cc0ac8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7fa3d27f0640 1 -- 172.21.15.60:0/503245253 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa3cc099e60 con 0x7fa3cc06e7b0 2024-05-09T05:00:35.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f2a5d3b4640 1 --2- 172.21.15.60:0/4009042441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a48003c30 0x7f2a500dad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f2a5d3b4640 1 -- 172.21.15.60:0/4009042441 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a500db570 con 0x7f2a500d1fa0 2024-05-09T05:00:35.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7f35b2d9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35b40b2670 0x7f35b41550a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.60:35146/0 (socket says 172.21.15.60:35146) 2024-05-09T05:00:35.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f2a5cbb3640 1 --2- 172.21.15.60:0/4009042441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a48003c30 0x7f2a500dad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:35.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.069+0000 7f35b2d9d640 1 -- 172.21.15.60:0/2661281499 learned_addr learned my addr 172.21.15.60:0/2661281499 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:35.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f35b259c640 1 --2- 172.21.15.60:0/2661281499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35b40688d0 0x7f35b406dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f44ba4ee640 1 --2- 172.21.15.60:0/1917377462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b41637d0 0x7f44b406fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f35b1d9b640 1 --2- 172.21.15.60:0/2661281499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35b40ad040 0x7f35b406e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f35b2d9d640 1 -- 172.21.15.60:0/2661281499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35b40688d0 msgr2=0x7f35b406dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f35b2d9d640 1 --2- 172.21.15.60:0/2661281499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35b40688d0 0x7f35b406dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f44b94ec640 1 -- 172.21.15.60:0/1917377462 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44b40aecf0 msgr2=0x7f44b414e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f44b9ced640 1 --2- 172.21.15.60:0/1917377462 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44b40aecf0 0x7f44b414e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f44b94ec640 1 --2- 172.21.15.60:0/1917377462 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44b40aecf0 0x7f44b414e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f44b94ec640 1 -- 172.21.15.60:0/1917377462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b41637d0 msgr2=0x7f44b406fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f44b94ec640 1 --2- 172.21.15.60:0/1917377462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b41637d0 0x7f44b406fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f44b94ec640 1 -- 172.21.15.60:0/1917377462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b41666d0 con 0x7f44b40b0980 2024-05-09T05:00:35.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f35b2d9d640 1 -- 172.21.15.60:0/2661281499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35b40ad040 msgr2=0x7f35b406e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f35b2d9d640 1 --2- 172.21.15.60:0/2661281499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35b40ad040 0x7f35b406e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7eff8a7f1640 1 -- 172.21.15.60:0/1526119212 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff8406c670 msgr2=0x7eff840af000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7eff8a7f1640 1 --2- 172.21.15.60:0/1526119212 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff8406c670 0x7eff840af000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f2a5d3b4640 1 --2- 172.21.15.60:0/4009042441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a500d1fa0 0x7f2a500d2420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2a58069b50 tx=0x7f2a58069870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.071+0000 7eff8a7f1640 1 -- 172.21.15.60:0/1526119212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff840d0730 msgr2=0x7eff84152010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.071+0000 7eff8a7f1640 1 --2- 172.21.15.60:0/1526119212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff840d0730 0x7eff84152010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f35b2d9d640 1 -- 172.21.15.60:0/2661281499 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b40aa650 con 0x7f35b40b2670 2024-05-09T05:00:35.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.071+0000 7f2a557fa640 1 -- 172.21.15.60:0/4009042441 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a58074f40 con 0x7f2a500d1fa0 2024-05-09T05:00:35.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.071+0000 7f2a5dbb5640 1 -- 172.21.15.60:0/4009042441 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a500db850 con 0x7f2a500d1fa0 2024-05-09T05:00:35.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.071+0000 7f2a5dbb5640 1 -- 172.21.15.60:0/4009042441 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a500bc4f0 con 0x7f2a500d1fa0 2024-05-09T05:00:35.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.071+0000 7eff8a7f1640 1 -- 172.21.15.60:0/1526119212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff84152550 con 0x7eff8415b5b0 2024-05-09T05:00:35.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f35b259c640 1 --2- 172.21.15.60:0/2661281499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35b40688d0 0x7f35b406dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:35.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f44ba4ee640 1 --2- 172.21.15.60:0/1917377462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b41637d0 0x7f44b406fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:35.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.070+0000 7f44b9ced640 1 --2- 172.21.15.60:0/1917377462 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44b40aecf0 0x7f44b414e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:35.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.071+0000 7f44b94ec640 1 --2- 172.21.15.60:0/1917377462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44b40b0980 0x7f44b4164700 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f44a40098c0 tx=0x7f44a40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.071+0000 7f35b2d9d640 1 --2- 172.21.15.60:0/2661281499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35b40b2670 0x7f35b41550a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f35a800bed0 tx=0x7f35a8013480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.071+0000 7fa3cbfff640 1 -- 172.21.15.60:0/503245253 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3cc15b5d0 msgr2=0x7fa3cc0ace30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.071+0000 7fa3cbfff640 1 --2- 172.21.15.60:0/503245253 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3cc15b5d0 0x7fa3cc0ace30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.071+0000 7fa3cbfff640 1 -- 172.21.15.60:0/503245253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3cc097de0 msgr2=0x7fa3cc0ac8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.071+0000 7fa3cbfff640 1 --2- 172.21.15.60:0/503245253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3cc097de0 0x7fa3cc0ac8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.071+0000 7fa3cbfff640 1 -- 172.21.15.60:0/503245253 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3cc151f30 con 0x7fa3cc06e7b0 2024-05-09T05:00:35.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.071+0000 7eff8a7f1640 1 --2- 172.21.15.60:0/1526119212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff8415b5b0 0x7eff840a75d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7eff8000e880 tx=0x7eff8000ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.072+0000 7f35a2ffd640 1 -- 172.21.15.60:0/2661281499 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a80113e0 con 0x7f35b40b2670 2024-05-09T05:00:35.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.072+0000 7eff7a7fc640 1 -- 172.21.15.60:0/1526119212 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff80017070 con 0x7eff8415b5b0 2024-05-09T05:00:35.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.072+0000 7fa3cbfff640 1 --2- 172.21.15.60:0/503245253 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3cc06e7b0 0x7fa3cc0cf790 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc00e690 tx=0x7fa3bc00eb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.072+0000 7f35b8e1d640 1 -- 172.21.15.60:0/2661281499 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35b40ab600 con 0x7f35b40b2670 2024-05-09T05:00:35.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.072+0000 7f35b8e1d640 1 -- 172.21.15.60:0/2661281499 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f35b40b1b80 con 0x7f35b40b2670 2024-05-09T05:00:35.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.071+0000 7f2a557fa640 1 -- 172.21.15.60:0/4009042441 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a58051040 con 0x7f2a500d1fa0 2024-05-09T05:00:35.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.071+0000 7f2a557fa640 1 -- 172.21.15.60:0/4009042441 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a58072350 con 0x7f2a500d1fa0 2024-05-09T05:00:35.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.072+0000 7eff8aff2640 1 -- 172.21.15.60:0/1526119212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff84150f70 con 0x7eff8415b5b0 2024-05-09T05:00:35.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.072+0000 7fa3c8ff9640 1 -- 172.21.15.60:0/503245253 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc016070 con 0x7fa3cc06e7b0 2024-05-09T05:00:35.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.072+0000 7fa3c8ff9640 1 -- 172.21.15.60:0/503245253 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3bc00ca30 con 0x7fa3cc06e7b0 2024-05-09T05:00:35.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.072+0000 7fa3c8ff9640 1 -- 172.21.15.60:0/503245253 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc00b040 con 0x7fa3cc06e7b0 2024-05-09T05:00:35.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.072+0000 7fa3d27f0640 1 -- 172.21.15.60:0/503245253 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3cc0a7120 con 0x7fa3cc06e7b0 2024-05-09T05:00:35.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.072+0000 7eff8aff2640 1 -- 172.21.15.60:0/1526119212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff84151530 con 0x7eff8415b5b0 2024-05-09T05:00:35.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.072+0000 7f44aa7fc640 1 -- 172.21.15.60:0/1917377462 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a4048040 con 0x7f44b40b0980 2024-05-09T05:00:35.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.072+0000 7f44aa7fc640 1 -- 172.21.15.60:0/1917377462 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44a4041500 con 0x7f44b40b0980 2024-05-09T05:00:35.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.072+0000 7f44aa7fc640 1 -- 172.21.15.60:0/1917377462 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a40418d0 con 0x7f44b40b0980 2024-05-09T05:00:35.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.072+0000 7f44bbf51640 1 -- 172.21.15.60:0/1917377462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44b414ee80 con 0x7f44b40b0980 2024-05-09T05:00:35.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.072+0000 7f44bbf51640 1 -- 172.21.15.60:0/1917377462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44b4165910 con 0x7f44b40b0980 2024-05-09T05:00:35.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.072+0000 7fa3d27f0640 1 -- 172.21.15.60:0/503245253 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3cc0afb70 con 0x7fa3cc06e7b0 2024-05-09T05:00:35.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.072+0000 7f35a2ffd640 1 -- 172.21.15.60:0/2661281499 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35a8011580 con 0x7f35b40b2670 2024-05-09T05:00:35.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.073+0000 7f35a2ffd640 1 -- 172.21.15.60:0/2661281499 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a80105c0 con 0x7f35b40b2670 2024-05-09T05:00:35.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.073+0000 7f2a557fa640 1 -- 172.21.15.60:0/4009042441 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2a580b58e0 con 0x7f2a500d1fa0 2024-05-09T05:00:35.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.074+0000 7f2a5dbb5640 1 -- 172.21.15.60:0/4009042441 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a50005390 con 0x7f2a500d1fa0 2024-05-09T05:00:35.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.074+0000 7f44bbf51640 1 -- 172.21.15.60:0/1917377462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f449c005290 con 0x7f44b40b0980 2024-05-09T05:00:35.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.074+0000 7eff7a7fc640 1 -- 172.21.15.60:0/1526119212 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff80004020 con 0x7eff8415b5b0 2024-05-09T05:00:35.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.075+0000 7fa3abfff640 1 -- 172.21.15.60:0/503245253 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3cc151a40 con 0x7fa3cc06e7b0 2024-05-09T05:00:35.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.076+0000 7f2a557fa640 1 --2- 172.21.15.60:0/4009042441 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2a3c0763a0 0x7f2a3c078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.074+0000 7eff7a7fc640 1 -- 172.21.15.60:0/1526119212 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff800043f0 con 0x7eff8415b5b0 2024-05-09T05:00:35.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.076+0000 7f2a5cbb3640 1 --2- 172.21.15.60:0/4009042441 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2a3c0763a0 0x7f2a3c078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.076+0000 7f2a557fa640 1 -- 172.21.15.60:0/4009042441 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2a581147d0 con 0x7f2a500d1fa0 2024-05-09T05:00:35.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.077+0000 7f2a5cbb3640 1 --2- 172.21.15.60:0/4009042441 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2a3c0763a0 0x7f2a3c078860 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c00c190 tx=0x7f2a4c034040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.077+0000 7f35b8e1d640 1 -- 172.21.15.60:0/2661281499 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3598005290 con 0x7f35b40b2670 2024-05-09T05:00:35.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.077+0000 7eff67fff640 1 -- 172.21.15.60:0/1526119212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff840640c0 con 0x7eff8415b5b0 2024-05-09T05:00:35.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.077+0000 7fa3c8ff9640 1 -- 172.21.15.60:0/503245253 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa3bc019070 con 0x7fa3cc06e7b0 2024-05-09T05:00:35.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.078+0000 7fa3c8ff9640 1 --2- 172.21.15.60:0/503245253 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa3b80763d0 0x7fa3b8078890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.078+0000 7fa3c8ff9640 1 -- 172.21.15.60:0/503245253 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa3bc097ea0 con 0x7fa3cc06e7b0 2024-05-09T05:00:35.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.078+0000 7f44aa7fc640 1 -- 172.21.15.60:0/1917377462 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f44a4055020 con 0x7f44b40b0980 2024-05-09T05:00:35.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.078+0000 7eff7a7fc640 1 -- 172.21.15.60:0/1526119212 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7eff8000c940 con 0x7eff8415b5b0 2024-05-09T05:00:35.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.078+0000 7f35a2ffd640 1 -- 172.21.15.60:0/2661281499 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f35a8010760 con 0x7f35b40b2670 2024-05-09T05:00:35.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.079+0000 7eff7a7fc640 1 --2- 172.21.15.60:0/1526119212 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7eff680763a0 0x7eff68078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.079+0000 7fa3cb7fe640 1 --2- 172.21.15.60:0/503245253 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa3b80763d0 0x7fa3b8078890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.079+0000 7f35a2ffd640 1 --2- 172.21.15.60:0/2661281499 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f35940763d0 0x7f3594078890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.079+0000 7eff7a7fc640 1 -- 172.21.15.60:0/1526119212 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7eff80097610 con 0x7eff8415b5b0 2024-05-09T05:00:35.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.079+0000 7f44aa7fc640 1 --2- 172.21.15.60:0/1917377462 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f44980763a0 0x7f4498078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.079+0000 7f44aa7fc640 1 -- 172.21.15.60:0/1917377462 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f44a40c8c20 con 0x7f44b40b0980 2024-05-09T05:00:35.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.079+0000 7eff89ff0640 1 --2- 172.21.15.60:0/1526119212 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7eff680763a0 0x7eff68078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.079+0000 7f35a2ffd640 1 -- 172.21.15.60:0/2661281499 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f35a8097800 con 0x7f35b40b2670 2024-05-09T05:00:35.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.079+0000 7f35b259c640 1 --2- 172.21.15.60:0/2661281499 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f35940763d0 0x7f3594078890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.079+0000 7f44b9ced640 1 --2- 172.21.15.60:0/1917377462 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f44980763a0 0x7f4498078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.080+0000 7fa3cb7fe640 1 --2- 172.21.15.60:0/503245253 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa3b80763d0 0x7fa3b8078890 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc1527a0 tx=0x7fa3c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.080+0000 7eff89ff0640 1 --2- 172.21.15.60:0/1526119212 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7eff680763a0 0x7eff68078860 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7eff7c02a6f0 tx=0x7eff7c009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.080+0000 7f44b9ced640 1 --2- 172.21.15.60:0/1917377462 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f44980763a0 0x7f4498078860 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f44b0000f60 tx=0x7f44b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.080+0000 7f35b259c640 1 --2- 172.21.15.60:0/2661281499 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f35940763d0 0x7f3594078890 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f35b40b4370 tx=0x7f35ac044040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.081+0000 7fa3c8ff9640 1 -- 172.21.15.60:0/503245253 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa3bc061530 con 0x7fa3cc06e7b0 2024-05-09T05:00:35.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.083+0000 7f44aa7fc640 1 -- 172.21.15.60:0/1917377462 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f44a4092220 con 0x7f44b40b0980 2024-05-09T05:00:35.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.085+0000 7f2a557fa640 1 -- 172.21.15.60:0/4009042441 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2a580c3030 con 0x7f2a500d1fa0 2024-05-09T05:00:35.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.085+0000 7f35a2ffd640 1 -- 172.21.15.60:0/2661281499 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f35a8060e90 con 0x7f35b40b2670 2024-05-09T05:00:35.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.089+0000 7eff7a7fc640 1 -- 172.21.15.60:0/1526119212 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7eff80060650 con 0x7eff8415b5b0 2024-05-09T05:00:35.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.456+0000 7fb40759e640 1 -- 172.21.15.60:0/1121137233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fb408070130 con 0x7fb4080ae7a0 2024-05-09T05:00:35.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.457+0000 7fad6530a640 1 -- 172.21.15.60:0/2152312270 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fad58036d30 con 0x7fad580df1c0 2024-05-09T05:00:35.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.457+0000 7fb3f6ffd640 1 -- 172.21.15.60:0/1121137233 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb4000a40a0 con 0x7fb4080ae7a0 2024-05-09T05:00:35.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.457+0000 7f8872d78640 1 -- 172.21.15.60:0/2256816143 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f8854005850 con 0x7f886c0d0900 2024-05-09T05:00:35.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.458+0000 7fad477fe640 1 -- 172.21.15.60:0/2152312270 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fad600e0470 con 0x7fad580df1c0 2024-05-09T05:00:35.458 INFO:teuthology.orchestra.run.smithi060.stdout:55834574856 2024-05-09T05:00:35.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.459+0000 7f88627fc640 1 -- 172.21.15.60:0/2256816143 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f886800d040 con 0x7f886c0d0900 2024-05-09T05:00:35.459 INFO:teuthology.orchestra.run.smithi060.stdout:55834574855 2024-05-09T05:00:35.462 INFO:teuthology.orchestra.run.smithi060.stdout:55834574855 2024-05-09T05:00:35.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.461+0000 7fad6530a640 1 -- 172.21.15.60:0/2152312270 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fad4c0763a0 msgr2=0x7fad4c078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.461+0000 7fad6530a640 1 --2- 172.21.15.60:0/2152312270 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fad4c0763a0 0x7fad4c078860 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fad50000f60 tx=0x7fad50009040 comp rx=0 tx=0).stop 2024-05-09T05:00:35.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.461+0000 7fad6530a640 1 -- 172.21.15.60:0/2152312270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad580df1c0 msgr2=0x7fad58185420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.461+0000 7fad6530a640 1 --2- 172.21.15.60:0/2152312270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad580df1c0 0x7fad58185420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fad60050ac0 tx=0x7fad60051220 comp rx=0 tx=0).stop 2024-05-09T05:00:35.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.461+0000 7fad6530a640 1 -- 172.21.15.60:0/2152312270 shutdown_connections 2024-05-09T05:00:35.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.461+0000 7fad6530a640 1 --2- 172.21.15.60:0/2152312270 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fad4c0763a0 0x7fad4c078860 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.461+0000 7fad6530a640 1 --2- 172.21.15.60:0/2152312270 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fad580dfcb0 0x7fad5802b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.461+0000 7fad6530a640 1 --2- 172.21.15.60:0/2152312270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad580df1c0 0x7fad58185420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.461+0000 7fad6530a640 1 --2- 172.21.15.60:0/2152312270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad58054690 0x7fad5804e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.461+0000 7fad6530a640 1 -- 172.21.15.60:0/2152312270 >> 172.21.15.60:0/2152312270 conn(0x7fad58009fe0 msgr2=0x7fad58039120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:35.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.462+0000 7fad6530a640 1 -- 172.21.15.60:0/2152312270 shutdown_connections 2024-05-09T05:00:35.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.462+0000 7fad6530a640 1 -- 172.21.15.60:0/2152312270 wait complete. 2024-05-09T05:00:35.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.462+0000 7fb40759e640 1 -- 172.21.15.60:0/1121137233 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb3e8076350 msgr2=0x7fb3e8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.462+0000 7fb40759e640 1 --2- 172.21.15.60:0/1121137233 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb3e8076350 0x7fb3e8078810 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc009950 tx=0x7fb3fc008040 comp rx=0 tx=0).stop 2024-05-09T05:00:35.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.462+0000 7fb40759e640 1 -- 172.21.15.60:0/1121137233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4080ae7a0 msgr2=0x7fb408161980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.462+0000 7fb40759e640 1 --2- 172.21.15.60:0/1121137233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4080ae7a0 0x7fb408161980 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb40002abe0 tx=0x7fb400004a40 comp rx=0 tx=0).stop 2024-05-09T05:00:35.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.462+0000 7fb40759e640 1 -- 172.21.15.60:0/1121137233 shutdown_connections 2024-05-09T05:00:35.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.462+0000 7fb40759e640 1 --2- 172.21.15.60:0/1121137233 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb3e8076350 0x7fb3e8078810 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.462+0000 7fb40759e640 1 --2- 172.21.15.60:0/1121137233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4080ae7a0 0x7fb408161980 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.462+0000 7fb40759e640 1 --2- 172.21.15.60:0/1121137233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb408073040 0x7fb4080ae260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.462+0000 7fb40759e640 1 --2- 172.21.15.60:0/1121137233 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb40806fcb0 0x7fb4080cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.462+0000 7fb40759e640 1 -- 172.21.15.60:0/1121137233 >> 172.21.15.60:0/1121137233 conn(0x7fb4080b4000 msgr2=0x7fb4080aa940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:35.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.462+0000 7f8872d78640 1 -- 172.21.15.60:0/2256816143 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8850076350 msgr2=0x7f8850078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.463+0000 7f8872d78640 1 --2- 172.21.15.60:0/2256816143 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8850076350 0x7f8850078810 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f885c006fd0 tx=0x7f885c009060 comp rx=0 tx=0).stop 2024-05-09T05:00:35.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.463+0000 7f8872d78640 1 -- 172.21.15.60:0/2256816143 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f886c0d0900 msgr2=0x7f886c14f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.463+0000 7f8872d78640 1 --2- 172.21.15.60:0/2256816143 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f886c0d0900 0x7f886c14f910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f886800b450 tx=0x7f886800b910 comp rx=0 tx=0).stop 2024-05-09T05:00:35.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.463+0000 7f8872d78640 1 -- 172.21.15.60:0/2256816143 shutdown_connections 2024-05-09T05:00:35.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.463+0000 7f8872d78640 1 --2- 172.21.15.60:0/2256816143 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8850076350 0x7f8850078810 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.463+0000 7f8872d78640 1 --2- 172.21.15.60:0/2256816143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f886c15b520 0x7f886c0a9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.463+0000 7f8872d78640 1 --2- 172.21.15.60:0/2256816143 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f886c0d0900 0x7f886c14f910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.463+0000 7fb40759e640 1 -- 172.21.15.60:0/1121137233 shutdown_connections 2024-05-09T05:00:35.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.463+0000 7fb40759e640 1 -- 172.21.15.60:0/1121137233 wait complete. 2024-05-09T05:00:35.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.463+0000 7f8872d78640 1 --2- 172.21.15.60:0/2256816143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f886c06c670 0x7f886c0ae970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.463+0000 7f8872d78640 1 -- 172.21.15.60:0/2256816143 >> 172.21.15.60:0/2256816143 conn(0x7f886c0b17d0 msgr2=0x7f886c14e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:35.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.463+0000 7f8872d78640 1 -- 172.21.15.60:0/2256816143 shutdown_connections 2024-05-09T05:00:35.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.463+0000 7f8872d78640 1 -- 172.21.15.60:0/2256816143 wait complete. 2024-05-09T05:00:35.483 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.5 2024-05-09T05:00:35.483 DEBUG:teuthology.parallel:result is None 2024-05-09T05:00:35.484 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2024-05-09T05:00:35.484 DEBUG:teuthology.parallel:result is None 2024-05-09T05:00:35.485 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2024-05-09T05:00:35.485 DEBUG:teuthology.parallel:result is None 2024-05-09T05:00:35.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.500+0000 7f44bbf51640 1 -- 172.21.15.60:0/1917377462 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f449c005580 con 0x7f44b40b0980 2024-05-09T05:00:35.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.500+0000 7f35b8e1d640 1 -- 172.21.15.60:0/2661281499 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f3598005580 con 0x7f35b40b2670 2024-05-09T05:00:35.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.501+0000 7f35a2ffd640 1 -- 172.21.15.60:0/2661281499 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f35a8060790 con 0x7f35b40b2670 2024-05-09T05:00:35.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.501+0000 7f44aa7fc640 1 -- 172.21.15.60:0/1917377462 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f44a4091b20 con 0x7f44b40b0980 2024-05-09T05:00:35.502 INFO:teuthology.orchestra.run.smithi060.stdout:55834574855 2024-05-09T05:00:35.502 INFO:teuthology.orchestra.run.smithi060.stdout:55834574855 2024-05-09T05:00:35.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.503+0000 7eff67fff640 1 -- 172.21.15.60:0/1526119212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7eff840b3520 con 0x7eff8415b5b0 2024-05-09T05:00:35.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.503+0000 7f35b8e1d640 1 -- 172.21.15.60:0/2661281499 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f35940763d0 msgr2=0x7f3594078890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.503+0000 7f35b8e1d640 1 --2- 172.21.15.60:0/2661281499 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f35940763d0 0x7f3594078890 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f35b40b4370 tx=0x7f35ac044040 comp rx=0 tx=0).stop 2024-05-09T05:00:35.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.503+0000 7f35b8e1d640 1 -- 172.21.15.60:0/2661281499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35b40b2670 msgr2=0x7f35b41550a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.503+0000 7f35b8e1d640 1 --2- 172.21.15.60:0/2661281499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35b40b2670 0x7f35b41550a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f35a800bed0 tx=0x7f35a8013480 comp rx=0 tx=0).stop 2024-05-09T05:00:35.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.503+0000 7f35b8e1d640 1 -- 172.21.15.60:0/2661281499 shutdown_connections 2024-05-09T05:00:35.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.503+0000 7f35b8e1d640 1 --2- 172.21.15.60:0/2661281499 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f35940763d0 0x7f3594078890 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.503+0000 7f35b8e1d640 1 --2- 172.21.15.60:0/2661281499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f35b40b2670 0x7f35b41550a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.503+0000 7f35b8e1d640 1 --2- 172.21.15.60:0/2661281499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35b40ad040 0x7f35b406e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.503+0000 7f35b8e1d640 1 --2- 172.21.15.60:0/2661281499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35b40688d0 0x7f35b406dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.503+0000 7f35b8e1d640 1 -- 172.21.15.60:0/2661281499 >> 172.21.15.60:0/2661281499 conn(0x7f35b40684c0 msgr2=0x7f35b415d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:35.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.503+0000 7f35b8e1d640 1 -- 172.21.15.60:0/2661281499 shutdown_connections 2024-05-09T05:00:35.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.503+0000 7f35b8e1d640 1 -- 172.21.15.60:0/2661281499 wait complete. 2024-05-09T05:00:35.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.503+0000 7eff7a7fc640 1 -- 172.21.15.60:0/1526119212 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7eff80064950 con 0x7eff8415b5b0 2024-05-09T05:00:35.507 INFO:teuthology.orchestra.run.smithi060.stdout:55834574855 2024-05-09T05:00:35.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.504+0000 7f44bbf51640 1 -- 172.21.15.60:0/1917377462 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f44980763a0 msgr2=0x7f4498078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.504+0000 7f44bbf51640 1 --2- 172.21.15.60:0/1917377462 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f44980763a0 0x7f4498078860 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f44b0000f60 tx=0x7f44b0009040 comp rx=0 tx=0).stop 2024-05-09T05:00:35.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.504+0000 7f44bbf51640 1 -- 172.21.15.60:0/1917377462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44b40b0980 msgr2=0x7f44b4164700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.504+0000 7f44bbf51640 1 --2- 172.21.15.60:0/1917377462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44b40b0980 0x7f44b4164700 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f44a40098c0 tx=0x7f44a40046c0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.504+0000 7f44bbf51640 1 -- 172.21.15.60:0/1917377462 shutdown_connections 2024-05-09T05:00:35.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.504+0000 7f44bbf51640 1 --2- 172.21.15.60:0/1917377462 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f44980763a0 0x7f4498078860 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.504+0000 7f44bbf51640 1 --2- 172.21.15.60:0/1917377462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b41637d0 0x7f44b406fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.504+0000 7f44bbf51640 1 --2- 172.21.15.60:0/1917377462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44b40b0980 0x7f44b4164700 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.504+0000 7f44bbf51640 1 --2- 172.21.15.60:0/1917377462 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f44b40aecf0 0x7f44b414e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.504+0000 7f44bbf51640 1 -- 172.21.15.60:0/1917377462 >> 172.21.15.60:0/1917377462 conn(0x7f44b4073240 msgr2=0x7f44b40b0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:35.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.504+0000 7f44bbf51640 1 -- 172.21.15.60:0/1917377462 shutdown_connections 2024-05-09T05:00:35.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.504+0000 7f44bbf51640 1 -- 172.21.15.60:0/1917377462 wait complete. 2024-05-09T05:00:35.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.506+0000 7eff67fff640 1 -- 172.21.15.60:0/1526119212 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7eff680763a0 msgr2=0x7eff68078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.506+0000 7eff67fff640 1 --2- 172.21.15.60:0/1526119212 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7eff680763a0 0x7eff68078860 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7eff7c02a6f0 tx=0x7eff7c009000 comp rx=0 tx=0).stop 2024-05-09T05:00:35.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.507+0000 7eff67fff640 1 -- 172.21.15.60:0/1526119212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff8415b5b0 msgr2=0x7eff840a75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.507+0000 7eff67fff640 1 --2- 172.21.15.60:0/1526119212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff8415b5b0 0x7eff840a75d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7eff8000e880 tx=0x7eff8000ed40 comp rx=0 tx=0).stop 2024-05-09T05:00:35.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.507+0000 7eff67fff640 1 -- 172.21.15.60:0/1526119212 shutdown_connections 2024-05-09T05:00:35.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.507+0000 7eff67fff640 1 --2- 172.21.15.60:0/1526119212 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7eff680763a0 0x7eff68078860 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.507+0000 7eff67fff640 1 --2- 172.21.15.60:0/1526119212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff8415b5b0 0x7eff840a75d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.507+0000 7eff67fff640 1 --2- 172.21.15.60:0/1526119212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff840d0730 0x7eff84152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.507+0000 7eff67fff640 1 --2- 172.21.15.60:0/1526119212 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7eff8406c670 0x7eff840af000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.507+0000 7eff67fff640 1 -- 172.21.15.60:0/1526119212 >> 172.21.15.60:0/1526119212 conn(0x7eff840b17d0 msgr2=0x7eff840b3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:35.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.507+0000 7eff67fff640 1 -- 172.21.15.60:0/1526119212 shutdown_connections 2024-05-09T05:00:35.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.507+0000 7eff67fff640 1 -- 172.21.15.60:0/1526119212 wait complete. 2024-05-09T05:00:35.518 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2024-05-09T05:00:35.518 DEBUG:teuthology.parallel:result is None 2024-05-09T05:00:35.521 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.2 2024-05-09T05:00:35.521 DEBUG:teuthology.parallel:result is None 2024-05-09T05:00:35.523 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2024-05-09T05:00:35.523 DEBUG:teuthology.parallel:result is None 2024-05-09T05:00:35.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.530+0000 7fa3abfff640 1 -- 172.21.15.60:0/503245253 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fa3cc051040 con 0x7fa3cc06e7b0 2024-05-09T05:00:35.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.530+0000 7fa3c8ff9640 1 -- 172.21.15.60:0/503245253 <== mon.2 v2:172.21.15.114:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa3bc060e30 con 0x7fa3cc06e7b0 2024-05-09T05:00:35.531 INFO:teuthology.orchestra.run.smithi060.stdout:55834574856 2024-05-09T05:00:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.533+0000 7fa3abfff640 1 -- 172.21.15.60:0/503245253 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa3b80763d0 msgr2=0x7fa3b8078890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.533+0000 7fa3abfff640 1 --2- 172.21.15.60:0/503245253 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa3b80763d0 0x7fa3b8078890 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc1527a0 tx=0x7fa3c0009040 comp rx=0 tx=0).stop 2024-05-09T05:00:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.533+0000 7fa3abfff640 1 -- 172.21.15.60:0/503245253 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3cc06e7b0 msgr2=0x7fa3cc0cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.533+0000 7fa3abfff640 1 --2- 172.21.15.60:0/503245253 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3cc06e7b0 0x7fa3cc0cf790 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc00e690 tx=0x7fa3bc00eb50 comp rx=0 tx=0).stop 2024-05-09T05:00:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.533+0000 7fa3abfff640 1 -- 172.21.15.60:0/503245253 shutdown_connections 2024-05-09T05:00:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.533+0000 7fa3abfff640 1 --2- 172.21.15.60:0/503245253 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa3b80763d0 0x7fa3b8078890 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.533+0000 7fa3abfff640 1 --2- 172.21.15.60:0/503245253 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3cc15b5d0 0x7fa3cc0ace30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.533+0000 7fa3abfff640 1 --2- 172.21.15.60:0/503245253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3cc097de0 0x7fa3cc0ac8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.533+0000 7fa3abfff640 1 --2- 172.21.15.60:0/503245253 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3cc06e7b0 0x7fa3cc0cf790 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.533+0000 7fa3abfff640 1 -- 172.21.15.60:0/503245253 >> 172.21.15.60:0/503245253 conn(0x7fa3cc0972f0 msgr2=0x7fa3cc159f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.533+0000 7fa3abfff640 1 -- 172.21.15.60:0/503245253 shutdown_connections 2024-05-09T05:00:35.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.533+0000 7fa3abfff640 1 -- 172.21.15.60:0/503245253 wait complete. 2024-05-09T05:00:35.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.534+0000 7f2a5dbb5640 1 -- 172.21.15.60:0/4009042441 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f2a50004cb0 con 0x7f2a500d1fa0 2024-05-09T05:00:35.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.534+0000 7f2a557fa640 1 -- 172.21.15.60:0/4009042441 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2a580dddd0 con 0x7f2a500d1fa0 2024-05-09T05:00:35.535 INFO:teuthology.orchestra.run.smithi060.stdout:55834574856 2024-05-09T05:00:35.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.536+0000 7f2a5dbb5640 1 -- 172.21.15.60:0/4009042441 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2a3c0763a0 msgr2=0x7f2a3c078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.536+0000 7f2a5dbb5640 1 --2- 172.21.15.60:0/4009042441 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2a3c0763a0 0x7f2a3c078860 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c00c190 tx=0x7f2a4c034040 comp rx=0 tx=0).stop 2024-05-09T05:00:35.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.536+0000 7f2a5dbb5640 1 -- 172.21.15.60:0/4009042441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a500d1fa0 msgr2=0x7f2a500d2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.536+0000 7f2a5dbb5640 1 --2- 172.21.15.60:0/4009042441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a500d1fa0 0x7f2a500d2420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2a58069b50 tx=0x7f2a58069870 comp rx=0 tx=0).stop 2024-05-09T05:00:35.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.536+0000 7f2a5dbb5640 1 -- 172.21.15.60:0/4009042441 shutdown_connections 2024-05-09T05:00:35.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.536+0000 7f2a5dbb5640 1 --2- 172.21.15.60:0/4009042441 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2a3c0763a0 0x7f2a3c078860 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.536+0000 7f2a5dbb5640 1 --2- 172.21.15.60:0/4009042441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a500d1fa0 0x7f2a500d2420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.536+0000 7f2a5dbb5640 1 --2- 172.21.15.60:0/4009042441 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2a5004bb90 0x7f2a5004c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.536+0000 7f2a5dbb5640 1 --2- 172.21.15.60:0/4009042441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a48003c30 0x7f2a500dad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.536+0000 7f2a5dbb5640 1 -- 172.21.15.60:0/4009042441 >> 172.21.15.60:0/4009042441 conn(0x7f2a50009fb0 msgr2=0x7f2a50033800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:35.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.537+0000 7f2a5dbb5640 1 -- 172.21.15.60:0/4009042441 shutdown_connections 2024-05-09T05:00:35.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.537+0000 7f2a5dbb5640 1 -- 172.21.15.60:0/4009042441 wait complete. 2024-05-09T05:00:35.546 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2024-05-09T05:00:35.546 DEBUG:teuthology.parallel:result is None 2024-05-09T05:00:35.548 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.6 2024-05-09T05:00:35.548 DEBUG:teuthology.parallel:result is None 2024-05-09T05:00:35.549 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-09T05:00:35.549 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:00:35.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.692+0000 7fdb2ef9e640 1 Processor -- start 2024-05-09T05:00:35.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.692+0000 7fdb2ef9e640 1 -- start start 2024-05-09T05:00:35.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.693+0000 7fdb2ef9e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb2815d0e0 0x7fdb2806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.693+0000 7fdb2ef9e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb2809d080 0x7fdb2806f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.693+0000 7fdb2ef9e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb280b32b0 con 0x7fdb280aad80 2024-05-09T05:00:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.693+0000 7fdb2ef9e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fdb280b3590 con 0x7fdb2809d080 2024-05-09T05:00:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.693+0000 7fdb2ef9e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fdb280b3870 con 0x7fdb2815d0e0 2024-05-09T05:00:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.693+0000 7fdb2e79d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb2809d080 0x7fdb2806f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.693+0000 7fdb2e79d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb2809d080 0x7fdb2806f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35094/0 (socket says 172.21.15.60:35094) 2024-05-09T05:00:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.693+0000 7fdb2d79b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdb280aad80 0x7fdb28160c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38500/0 (socket says 172.21.15.60:38500) 2024-05-09T05:00:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.693+0000 7fdb2e79d640 1 -- 172.21.15.60:0/2312797503 learned_addr learned my addr 172.21.15.60:0/2312797503 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.694+0000 7fdb2df9c640 1 --2- 172.21.15.60:0/2312797503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb2815d0e0 0x7fdb2806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.694+0000 7fdb2e79d640 1 -- 172.21.15.60:0/2312797503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb2815d0e0 msgr2=0x7fdb2806ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.694+0000 7fdb2e79d640 1 --2- 172.21.15.60:0/2312797503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb2815d0e0 0x7fdb2806ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.694+0000 7fdb2e79d640 1 -- 172.21.15.60:0/2312797503 >> v1:172.21.15.60:6789/0 conn(0x7fdb280aad80 legacy=0x7fdb28160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.694+0000 7fdb2cf9a640 1 -- 172.21.15.60:0/2312797503 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1602601434 0 0) 0x7fdb280b32b0 con 0x7fdb280aad80 2024-05-09T05:00:35.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.694+0000 7fdb2e79d640 1 -- 172.21.15.60:0/2312797503 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb280b3b50 con 0x7fdb2809d080 2024-05-09T05:00:35.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.695+0000 7fdb2e79d640 1 --2- 172.21.15.60:0/2312797503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb2809d080 0x7fdb2806f1b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdb18009e50 tx=0x7fdb1802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1fe6e0ac5de8603 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.695+0000 7fdb2cf9a640 1 -- 172.21.15.60:0/2312797503 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb18036070 con 0x7fdb2809d080 2024-05-09T05:00:35.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.695+0000 7fdb2cf9a640 1 -- 172.21.15.60:0/2312797503 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb18031600 con 0x7fdb2809d080 2024-05-09T05:00:35.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.695+0000 7fdb2cf9a640 1 -- 172.21.15.60:0/2312797503 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb180319d0 con 0x7fdb2809d080 2024-05-09T05:00:35.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.695+0000 7fdb2ef9e640 1 -- 172.21.15.60:0/2312797503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb2809d080 msgr2=0x7fdb2806f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.695+0000 7fdb2ef9e640 1 --2- 172.21.15.60:0/2312797503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb2809d080 0x7fdb2806f1b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdb18009e50 tx=0x7fdb1802a680 comp rx=0 tx=0).stop 2024-05-09T05:00:35.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.695+0000 7fdb2ef9e640 1 -- 172.21.15.60:0/2312797503 shutdown_connections 2024-05-09T05:00:35.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.695+0000 7fdb2ef9e640 1 --2- 172.21.15.60:0/2312797503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb2809d080 0x7fdb2806f1b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.695+0000 7fdb2ef9e640 1 --2- 172.21.15.60:0/2312797503 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb2815d0e0 0x7fdb2806ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.695+0000 7fdb2ef9e640 1 -- 172.21.15.60:0/2312797503 >> 172.21.15.60:0/2312797503 conn(0x7fdb28071d60 msgr2=0x7fdb28072170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:35.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.695+0000 7fdb2ef9e640 1 -- 172.21.15.60:0/2312797503 shutdown_connections 2024-05-09T05:00:35.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.695+0000 7fdb2ef9e640 1 -- 172.21.15.60:0/2312797503 wait complete. 2024-05-09T05:00:35.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.697+0000 7fdb2ef9e640 1 Processor -- start 2024-05-09T05:00:35.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.697+0000 7fdb2ef9e640 1 -- start start 2024-05-09T05:00:35.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2ef9e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb2809d080 0x7fdb280ab1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2ef9e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb280aad80 0x7fdb280a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2ef9e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb2815d0e0 0x7fdb280adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2ef9e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdb280d2060 con 0x7fdb2815d0e0 2024-05-09T05:00:35.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2ef9e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2809cb30 con 0x7fdb2809d080 2024-05-09T05:00:35.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2ef9e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fdb280d21e0 con 0x7fdb280aad80 2024-05-09T05:00:35.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2e79d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb2815d0e0 0x7fdb280adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2e79d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb2815d0e0 0x7fdb280adbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35108/0 (socket says 172.21.15.60:35108) 2024-05-09T05:00:35.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2e79d640 1 -- 172.21.15.60:0/245940100 learned_addr learned my addr 172.21.15.60:0/245940100 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:35.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2d79b640 1 --2- 172.21.15.60:0/245940100 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb280aad80 0x7fdb280a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2df9c640 1 --2- 172.21.15.60:0/245940100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb2809d080 0x7fdb280ab1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2e79d640 1 -- 172.21.15.60:0/245940100 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb280aad80 msgr2=0x7fdb280a9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2e79d640 1 --2- 172.21.15.60:0/245940100 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb280aad80 0x7fdb280a9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2e79d640 1 -- 172.21.15.60:0/245940100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb2809d080 msgr2=0x7fdb280ab1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2e79d640 1 --2- 172.21.15.60:0/245940100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb2809d080 0x7fdb280ab1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2e79d640 1 -- 172.21.15.60:0/245940100 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb2815e8e0 con 0x7fdb2815d0e0 2024-05-09T05:00:35.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2df9c640 1 --2- 172.21.15.60:0/245940100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb2809d080 0x7fdb280ab1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:35.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2d79b640 1 --2- 172.21.15.60:0/245940100 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb280aad80 0x7fdb280a9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:35.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2e79d640 1 --2- 172.21.15.60:0/245940100 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb2815d0e0 0x7fdb280adbb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdb18009e50 tx=0x7fdb18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb267fc640 1 -- 172.21.15.60:0/245940100 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb18041020 con 0x7fdb2815d0e0 2024-05-09T05:00:35.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb267fc640 1 -- 172.21.15.60:0/245940100 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb18037040 con 0x7fdb2815d0e0 2024-05-09T05:00:35.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.698+0000 7fdb2ef9e640 1 -- 172.21.15.60:0/245940100 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb280ae0f0 con 0x7fdb2815d0e0 2024-05-09T05:00:35.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.699+0000 7fdb267fc640 1 -- 172.21.15.60:0/245940100 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb18036050 con 0x7fdb2815d0e0 2024-05-09T05:00:35.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.699+0000 7fdb2ef9e640 1 -- 172.21.15.60:0/245940100 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb2815ee60 con 0x7fdb2815d0e0 2024-05-09T05:00:35.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.700+0000 7fdb2ef9e640 1 -- 172.21.15.60:0/245940100 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb280640c0 con 0x7fdb2815d0e0 2024-05-09T05:00:35.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.700+0000 7fdb267fc640 1 -- 172.21.15.60:0/245940100 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdb18007c30 con 0x7fdb2815d0e0 2024-05-09T05:00:35.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.701+0000 7fdb267fc640 1 --2- 172.21.15.60:0/245940100 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fdb140762d0 0x7fdb14078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:35.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.701+0000 7fdb267fc640 1 -- 172.21.15.60:0/245940100 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fdb180b80f0 con 0x7fdb2815d0e0 2024-05-09T05:00:35.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.701+0000 7fdb2df9c640 1 --2- 172.21.15.60:0/245940100 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fdb140762d0 0x7fdb14078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:35.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.702+0000 7fdb2df9c640 1 --2- 172.21.15.60:0/245940100 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fdb140762d0 0x7fdb14078790 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdb20006fd0 tx=0x7fdb20008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:35.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.705+0000 7fdb267fc640 1 -- 172.21.15.60:0/245940100 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fdb1804b060 con 0x7fdb2815d0e0 2024-05-09T05:00:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.892+0000 7fdb2ef9e640 1 -- 172.21.15.60:0/245940100 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdb280b2320 con 0x7fdb140762d0 2024-05-09T05:00:35.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.901+0000 7fdb267fc640 1 -- 172.21.15.60:0/245940100 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332523 (secure 0 0 0) 0x7fdb280b2320 con 0x7fdb140762d0 2024-05-09T05:00:35.905 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:35.907 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-05-09T05:00:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.904+0000 7fdb2ef9e640 1 -- 172.21.15.60:0/245940100 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fdb140762d0 msgr2=0x7fdb14078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.904+0000 7fdb2ef9e640 1 --2- 172.21.15.60:0/245940100 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fdb140762d0 0x7fdb14078790 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdb20006fd0 tx=0x7fdb20008040 comp rx=0 tx=0).stop 2024-05-09T05:00:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.904+0000 7fdb2ef9e640 1 -- 172.21.15.60:0/245940100 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb2815d0e0 msgr2=0x7fdb280adbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.904+0000 7fdb2ef9e640 1 --2- 172.21.15.60:0/245940100 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb2815d0e0 0x7fdb280adbb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdb18009e50 tx=0x7fdb18004a40 comp rx=0 tx=0).stop 2024-05-09T05:00:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.904+0000 7fdb2ef9e640 1 -- 172.21.15.60:0/245940100 shutdown_connections 2024-05-09T05:00:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.904+0000 7fdb2ef9e640 1 --2- 172.21.15.60:0/245940100 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fdb140762d0 0x7fdb14078790 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.904+0000 7fdb2ef9e640 1 --2- 172.21.15.60:0/245940100 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb2815d0e0 0x7fdb280adbb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.904+0000 7fdb2ef9e640 1 --2- 172.21.15.60:0/245940100 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdb280aad80 0x7fdb280a9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.905+0000 7fdb2ef9e640 1 --2- 172.21.15.60:0/245940100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb2809d080 0x7fdb280ab1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.905+0000 7fdb2ef9e640 1 -- 172.21.15.60:0/245940100 >> 172.21.15.60:0/245940100 conn(0x7fdb28071d60 msgr2=0x7fdb280991c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.905+0000 7fdb2ef9e640 1 -- 172.21.15.60:0/245940100 shutdown_connections 2024-05-09T05:00:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:35.905+0000 7fdb2ef9e640 1 -- 172.21.15.60:0/245940100 wait complete. 2024-05-09T05:00:35.916 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-05-09T05:00:34.916031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223524,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751196,"statfs":{"total":773094113280,"available":772865224704,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003753"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624117+0000","last_change":"2024-05-09T05:00:11.849304+0000","last_active":"2024-05-09T05:00:19.624117+0000","last_peered":"2024-05-09T05:00:19.624117+0000","last_clean":"2024-05-09T05:00:19.624117+0000","last_became_active":"2024-05-09T05:00:11.848977+0000","last_became_peered":"2024-05-09T05:00:11.848977+0000","last_unstale":"2024-05-09T05:00:19.624117+0000","last_undegraded":"2024-05-09T05:00:19.624117+0000","last_fullsized":"2024-05-09T05:00:19.624117+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:48:26.640184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652587+0000","last_change":"2024-05-09T05:00:12.853461+0000","last_active":"2024-05-09T05:00:18.652587+0000","last_peered":"2024-05-09T05:00:18.652587+0000","last_clean":"2024-05-09T05:00:18.652587+0000","last_became_active":"2024-05-09T05:00:12.853171+0000","last_became_peered":"2024-05-09T05:00:12.853171+0000","last_unstale":"2024-05-09T05:00:18.652587+0000","last_undegraded":"2024-05-09T05:00:18.652587+0000","last_fullsized":"2024-05-09T05:00:18.652587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:01:08.488591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652158+0000","last_change":"2024-05-09T05:00:16.883712+0000","last_active":"2024-05-09T05:00:18.652158+0000","last_peered":"2024-05-09T05:00:18.652158+0000","last_clean":"2024-05-09T05:00:18.652158+0000","last_became_active":"2024-05-09T05:00:16.883004+0000","last_became_peered":"2024-05-09T05:00:16.883004+0000","last_unstale":"2024-05-09T05:00:18.652158+0000","last_undegraded":"2024-05-09T05:00:18.652158+0000","last_fullsized":"2024-05-09T05:00:18.652158+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:27.371200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652845+0000","last_change":"2024-05-09T05:00:11.839791+0000","last_active":"2024-05-09T05:00:18.652845+0000","last_peered":"2024-05-09T05:00:18.652845+0000","last_clean":"2024-05-09T05:00:18.652845+0000","last_became_active":"2024-05-09T05:00:11.839531+0000","last_became_peered":"2024-05-09T05:00:11.839531+0000","last_unstale":"2024-05-09T05:00:18.652845+0000","last_undegraded":"2024-05-09T05:00:18.652845+0000","last_fullsized":"2024-05-09T05:00:18.652845+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:23.679256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623978+0000","last_change":"2024-05-09T05:00:12.844866+0000","last_active":"2024-05-09T05:00:19.623978+0000","last_peered":"2024-05-09T05:00:19.623978+0000","last_clean":"2024-05-09T05:00:19.623978+0000","last_became_active":"2024-05-09T05:00:12.844015+0000","last_became_peered":"2024-05-09T05:00:12.844015+0000","last_unstale":"2024-05-09T05:00:19.623978+0000","last_undegraded":"2024-05-09T05:00:19.623978+0000","last_fullsized":"2024-05-09T05:00:19.623978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881734+0000","last_change":"2024-05-09T05:00:16.878021+0000","last_active":"2024-05-09T05:00:19.881734+0000","last_peered":"2024-05-09T05:00:19.881734+0000","last_clean":"2024-05-09T05:00:19.881734+0000","last_became_active":"2024-05-09T05:00:16.877322+0000","last_became_peered":"2024-05-09T05:00:16.877322+0000","last_unstale":"2024-05-09T05:00:19.881734+0000","last_undegraded":"2024-05-09T05:00:19.881734+0000","last_fullsized":"2024-05-09T05:00:19.881734+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:34.719408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270583+0000","last_change":"2024-05-09T05:00:11.852994+0000","last_active":"2024-05-09T05:00:20.270583+0000","last_peered":"2024-05-09T05:00:20.270583+0000","last_clean":"2024-05-09T05:00:20.270583+0000","last_became_active":"2024-05-09T05:00:11.852737+0000","last_became_peered":"2024-05-09T05:00:11.852737+0000","last_unstale":"2024-05-09T05:00:20.270583+0000","last_undegraded":"2024-05-09T05:00:20.270583+0000","last_fullsized":"2024-05-09T05:00:20.270583+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:39.794811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970762+0000","last_change":"2024-05-09T05:00:12.845198+0000","last_active":"2024-05-09T05:00:19.970762+0000","last_peered":"2024-05-09T05:00:19.970762+0000","last_clean":"2024-05-09T05:00:19.970762+0000","last_became_active":"2024-05-09T05:00:12.844003+0000","last_became_peered":"2024-05-09T05:00:12.844003+0000","last_unstale":"2024-05-09T05:00:19.970762+0000","last_undegraded":"2024-05-09T05:00:19.970762+0000","last_fullsized":"2024-05-09T05:00:19.970762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:45.314128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882179+0000","last_change":"2024-05-09T05:00:16.877475+0000","last_active":"2024-05-09T05:00:19.882179+0000","last_peered":"2024-05-09T05:00:19.882179+0000","last_clean":"2024-05-09T05:00:19.882179+0000","last_became_active":"2024-05-09T05:00:16.876947+0000","last_became_peered":"2024-05-09T05:00:16.876947+0000","last_unstale":"2024-05-09T05:00:19.882179+0000","last_undegraded":"2024-05-09T05:00:19.882179+0000","last_fullsized":"2024-05-09T05:00:19.882179+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:49.342204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271162+0000","last_change":"2024-05-09T05:00:11.853073+0000","last_active":"2024-05-09T05:00:20.271162+0000","last_peered":"2024-05-09T05:00:20.271162+0000","last_clean":"2024-05-09T05:00:20.271162+0000","last_became_active":"2024-05-09T05:00:11.852759+0000","last_became_peered":"2024-05-09T05:00:11.852759+0000","last_unstale":"2024-05-09T05:00:20.271162+0000","last_undegraded":"2024-05-09T05:00:20.271162+0000","last_fullsized":"2024-05-09T05:00:20.271162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:01.738771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970943+0000","last_change":"2024-05-09T05:00:12.844419+0000","last_active":"2024-05-09T05:00:19.970943+0000","last_peered":"2024-05-09T05:00:19.970943+0000","last_clean":"2024-05-09T05:00:19.970943+0000","last_became_active":"2024-05-09T05:00:12.843951+0000","last_became_peered":"2024-05-09T05:00:12.843951+0000","last_unstale":"2024-05-09T05:00:19.970943+0000","last_undegraded":"2024-05-09T05:00:19.970943+0000","last_fullsized":"2024-05-09T05:00:19.970943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:23.066586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881928+0000","last_change":"2024-05-09T05:00:16.875575+0000","last_active":"2024-05-09T05:00:19.881928+0000","last_peered":"2024-05-09T05:00:19.881928+0000","last_clean":"2024-05-09T05:00:19.881928+0000","last_became_active":"2024-05-09T05:00:16.874993+0000","last_became_peered":"2024-05-09T05:00:16.874993+0000","last_unstale":"2024-05-09T05:00:19.881928+0000","last_undegraded":"2024-05-09T05:00:19.881928+0000","last_fullsized":"2024-05-09T05:00:19.881928+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:34.414100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860116+0000","last_change":"2024-05-09T05:00:12.844310+0000","last_active":"2024-05-09T05:00:17.860116+0000","last_peered":"2024-05-09T05:00:17.860116+0000","last_clean":"2024-05-09T05:00:17.860116+0000","last_became_active":"2024-05-09T05:00:12.843870+0000","last_became_peered":"2024-05-09T05:00:12.843870+0000","last_unstale":"2024-05-09T05:00:17.860116+0000","last_undegraded":"2024-05-09T05:00:17.860116+0000","last_fullsized":"2024-05-09T05:00:17.860116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:19.080163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271207+0000","last_change":"2024-05-09T05:00:11.849654+0000","last_active":"2024-05-09T05:00:20.271207+0000","last_peered":"2024-05-09T05:00:20.271207+0000","last_clean":"2024-05-09T05:00:20.271207+0000","last_became_active":"2024-05-09T05:00:11.849112+0000","last_became_peered":"2024-05-09T05:00:11.849112+0000","last_unstale":"2024-05-09T05:00:20.271207+0000","last_undegraded":"2024-05-09T05:00:20.271207+0000","last_fullsized":"2024-05-09T05:00:20.271207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:04:53.801780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882282+0000","last_change":"2024-05-09T05:00:16.874883+0000","last_active":"2024-05-09T05:00:19.882282+0000","last_peered":"2024-05-09T05:00:19.882282+0000","last_clean":"2024-05-09T05:00:19.882282+0000","last_became_active":"2024-05-09T05:00:16.874390+0000","last_became_peered":"2024-05-09T05:00:16.874390+0000","last_unstale":"2024-05-09T05:00:19.882282+0000","last_undegraded":"2024-05-09T05:00:19.882282+0000","last_fullsized":"2024-05-09T05:00:19.882282+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:09.005203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652613+0000","last_change":"2024-05-09T05:00:12.855440+0000","last_active":"2024-05-09T05:00:18.652613+0000","last_peered":"2024-05-09T05:00:18.652613+0000","last_clean":"2024-05-09T05:00:18.652613+0000","last_became_active":"2024-05-09T05:00:12.854274+0000","last_became_peered":"2024-05-09T05:00:12.854274+0000","last_unstale":"2024-05-09T05:00:18.652613+0000","last_undegraded":"2024-05-09T05:00:18.652613+0000","last_fullsized":"2024-05-09T05:00:18.652613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:30.691332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270354+0000","last_change":"2024-05-09T05:00:11.842122+0000","last_active":"2024-05-09T05:00:20.270354+0000","last_peered":"2024-05-09T05:00:20.270354+0000","last_clean":"2024-05-09T05:00:20.270354+0000","last_became_active":"2024-05-09T05:00:11.840835+0000","last_became_peered":"2024-05-09T05:00:11.840835+0000","last_unstale":"2024-05-09T05:00:20.270354+0000","last_undegraded":"2024-05-09T05:00:20.270354+0000","last_fullsized":"2024-05-09T05:00:20.270354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:41:27.364045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882288+0000","last_change":"2024-05-09T05:00:16.875354+0000","last_active":"2024-05-09T05:00:19.882288+0000","last_peered":"2024-05-09T05:00:19.882288+0000","last_clean":"2024-05-09T05:00:19.882288+0000","last_became_active":"2024-05-09T05:00:16.874942+0000","last_became_peered":"2024-05-09T05:00:16.874942+0000","last_unstale":"2024-05-09T05:00:19.882288+0000","last_undegraded":"2024-05-09T05:00:19.882288+0000","last_fullsized":"2024-05-09T05:00:19.882288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:27.094662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652325+0000","last_change":"2024-05-09T05:00:12.853552+0000","last_active":"2024-05-09T05:00:18.652325+0000","last_peered":"2024-05-09T05:00:18.652325+0000","last_clean":"2024-05-09T05:00:18.652325+0000","last_became_active":"2024-05-09T05:00:12.853268+0000","last_became_peered":"2024-05-09T05:00:12.853268+0000","last_unstale":"2024-05-09T05:00:18.652325+0000","last_undegraded":"2024-05-09T05:00:18.652325+0000","last_fullsized":"2024-05-09T05:00:18.652325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:44.359533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270643+0000","last_change":"2024-05-09T05:00:11.848638+0000","last_active":"2024-05-09T05:00:20.270643+0000","last_peered":"2024-05-09T05:00:20.270643+0000","last_clean":"2024-05-09T05:00:20.270643+0000","last_became_active":"2024-05-09T05:00:11.848351+0000","last_became_peered":"2024-05-09T05:00:11.848351+0000","last_unstale":"2024-05-09T05:00:20.270643+0000","last_undegraded":"2024-05-09T05:00:20.270643+0000","last_fullsized":"2024-05-09T05:00:20.270643+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:01.738771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271321+0000","last_change":"2024-05-09T05:00:16.880744+0000","last_active":"2024-05-09T05:00:20.271321+0000","last_peered":"2024-05-09T05:00:20.271321+0000","last_clean":"2024-05-09T05:00:20.271321+0000","last_became_active":"2024-05-09T05:00:16.880373+0000","last_became_peered":"2024-05-09T05:00:16.880373+0000","last_unstale":"2024-05-09T05:00:20.271321+0000","last_undegraded":"2024-05-09T05:00:20.271321+0000","last_fullsized":"2024-05-09T05:00:20.271321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:10.821888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882019+0000","last_change":"2024-05-09T05:00:12.835034+0000","last_active":"2024-05-09T05:00:19.882019+0000","last_peered":"2024-05-09T05:00:19.882019+0000","last_clean":"2024-05-09T05:00:19.882019+0000","last_became_active":"2024-05-09T05:00:12.834744+0000","last_became_peered":"2024-05-09T05:00:12.834744+0000","last_unstale":"2024-05-09T05:00:19.882019+0000","last_undegraded":"2024-05-09T05:00:19.882019+0000","last_fullsized":"2024-05-09T05:00:19.882019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:26.805203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270527+0000","last_change":"2024-05-09T05:00:11.848682+0000","last_active":"2024-05-09T05:00:20.270527+0000","last_peered":"2024-05-09T05:00:20.270527+0000","last_clean":"2024-05-09T05:00:20.270527+0000","last_became_active":"2024-05-09T05:00:11.848396+0000","last_became_peered":"2024-05-09T05:00:11.848396+0000","last_unstale":"2024-05-09T05:00:20.270527+0000","last_undegraded":"2024-05-09T05:00:20.270527+0000","last_fullsized":"2024-05-09T05:00:20.270527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:44:46.364938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882265+0000","last_change":"2024-05-09T05:00:16.877861+0000","last_active":"2024-05-09T05:00:19.882265+0000","last_peered":"2024-05-09T05:00:19.882265+0000","last_clean":"2024-05-09T05:00:19.882265+0000","last_became_active":"2024-05-09T05:00:16.877155+0000","last_became_peered":"2024-05-09T05:00:16.877155+0000","last_unstale":"2024-05-09T05:00:19.882265+0000","last_undegraded":"2024-05-09T05:00:19.882265+0000","last_fullsized":"2024-05-09T05:00:19.882265+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:01:12.516667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624129+0000","last_change":"2024-05-09T05:00:12.852207+0000","last_active":"2024-05-09T05:00:19.624129+0000","last_peered":"2024-05-09T05:00:19.624129+0000","last_clean":"2024-05-09T05:00:19.624129+0000","last_became_active":"2024-05-09T05:00:12.851927+0000","last_became_peered":"2024-05-09T05:00:12.851927+0000","last_unstale":"2024-05-09T05:00:19.624129+0000","last_undegraded":"2024-05-09T05:00:19.624129+0000","last_fullsized":"2024-05-09T05:00:19.624129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:44.359533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.861099+0000","last_change":"2024-05-09T05:00:11.849008+0000","last_active":"2024-05-09T05:00:17.861099+0000","last_peered":"2024-05-09T05:00:17.861099+0000","last_clean":"2024-05-09T05:00:17.861099+0000","last_became_active":"2024-05-09T05:00:11.848741+0000","last_became_peered":"2024-05-09T05:00:11.848741+0000","last_unstale":"2024-05-09T05:00:17.861099+0000","last_undegraded":"2024-05-09T05:00:17.861099+0000","last_fullsized":"2024-05-09T05:00:17.861099+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:39.794811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881982+0000","last_change":"2024-05-09T05:00:16.879683+0000","last_active":"2024-05-09T05:00:19.881982+0000","last_peered":"2024-05-09T05:00:19.881982+0000","last_clean":"2024-05-09T05:00:19.881982+0000","last_became_active":"2024-05-09T05:00:16.879337+0000","last_became_peered":"2024-05-09T05:00:16.879337+0000","last_unstale":"2024-05-09T05:00:19.881982+0000","last_undegraded":"2024-05-09T05:00:19.881982+0000","last_fullsized":"2024-05-09T05:00:19.881982+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:48.387609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652854+0000","last_change":"2024-05-09T05:00:12.852575+0000","last_active":"2024-05-09T05:00:18.652854+0000","last_peered":"2024-05-09T05:00:18.652854+0000","last_clean":"2024-05-09T05:00:18.652854+0000","last_became_active":"2024-05-09T05:00:12.852308+0000","last_became_peered":"2024-05-09T05:00:12.852308+0000","last_unstale":"2024-05-09T05:00:18.652854+0000","last_undegraded":"2024-05-09T05:00:18.652854+0000","last_fullsized":"2024-05-09T05:00:18.652854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270373+0000","last_change":"2024-05-09T05:00:11.839637+0000","last_active":"2024-05-09T05:00:20.270373+0000","last_peered":"2024-05-09T05:00:20.270373+0000","last_clean":"2024-05-09T05:00:20.270373+0000","last_became_active":"2024-05-09T05:00:11.839214+0000","last_became_peered":"2024-05-09T05:00:11.839214+0000","last_unstale":"2024-05-09T05:00:20.270373+0000","last_undegraded":"2024-05-09T05:00:20.270373+0000","last_fullsized":"2024-05-09T05:00:20.270373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:25.803010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652033+0000","last_change":"2024-05-09T05:00:16.883218+0000","last_active":"2024-05-09T05:00:18.652033+0000","last_peered":"2024-05-09T05:00:18.652033+0000","last_clean":"2024-05-09T05:00:18.652033+0000","last_became_active":"2024-05-09T05:00:16.882353+0000","last_became_peered":"2024-05-09T05:00:16.882353+0000","last_unstale":"2024-05-09T05:00:18.652033+0000","last_undegraded":"2024-05-09T05:00:18.652033+0000","last_fullsized":"2024-05-09T05:00:18.652033+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:02.171370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271136+0000","last_change":"2024-05-09T05:00:12.846656+0000","last_active":"2024-05-09T05:00:20.271136+0000","last_peered":"2024-05-09T05:00:20.271136+0000","last_clean":"2024-05-09T05:00:20.271136+0000","last_became_active":"2024-05-09T05:00:12.846359+0000","last_became_peered":"2024-05-09T05:00:12.846359+0000","last_unstale":"2024-05-09T05:00:20.271136+0000","last_undegraded":"2024-05-09T05:00:20.271136+0000","last_fullsized":"2024-05-09T05:00:20.271136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:49.330395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270574+0000","last_change":"2024-05-09T05:00:11.850170+0000","last_active":"2024-05-09T05:00:20.270574+0000","last_peered":"2024-05-09T05:00:20.270574+0000","last_clean":"2024-05-09T05:00:20.270574+0000","last_became_active":"2024-05-09T05:00:11.849866+0000","last_became_peered":"2024-05-09T05:00:11.849866+0000","last_unstale":"2024-05-09T05:00:20.270574+0000","last_undegraded":"2024-05-09T05:00:20.270574+0000","last_fullsized":"2024-05-09T05:00:20.270574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:27:57.420062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623671+0000","last_change":"2024-05-09T05:00:16.879852+0000","last_active":"2024-05-09T05:00:19.623671+0000","last_peered":"2024-05-09T05:00:19.623671+0000","last_clean":"2024-05-09T05:00:19.623671+0000","last_became_active":"2024-05-09T05:00:16.878219+0000","last_became_peered":"2024-05-09T05:00:16.878219+0000","last_unstale":"2024-05-09T05:00:19.623671+0000","last_undegraded":"2024-05-09T05:00:19.623671+0000","last_fullsized":"2024-05-09T05:00:19.623671+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:20.415639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.971002+0000","last_change":"2024-05-09T05:00:12.859874+0000","last_active":"2024-05-09T05:00:19.971002+0000","last_peered":"2024-05-09T05:00:19.971002+0000","last_clean":"2024-05-09T05:00:19.971002+0000","last_became_active":"2024-05-09T05:00:12.859620+0000","last_became_peered":"2024-05-09T05:00:12.859620+0000","last_unstale":"2024-05-09T05:00:19.971002+0000","last_undegraded":"2024-05-09T05:00:19.971002+0000","last_fullsized":"2024-05-09T05:00:19.971002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:04.977127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270358+0000","last_change":"2024-05-09T05:00:11.838593+0000","last_active":"2024-05-09T05:00:20.270358+0000","last_peered":"2024-05-09T05:00:20.270358+0000","last_clean":"2024-05-09T05:00:20.270358+0000","last_became_active":"2024-05-09T05:00:11.838314+0000","last_became_peered":"2024-05-09T05:00:11.838314+0000","last_unstale":"2024-05-09T05:00:20.270358+0000","last_undegraded":"2024-05-09T05:00:20.270358+0000","last_fullsized":"2024-05-09T05:00:20.270358+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:49.660089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270704+0000","last_change":"2024-05-09T05:00:16.876934+0000","last_active":"2024-05-09T05:00:20.270704+0000","last_peered":"2024-05-09T05:00:20.270704+0000","last_clean":"2024-05-09T05:00:20.270704+0000","last_became_active":"2024-05-09T05:00:16.876531+0000","last_became_peered":"2024-05-09T05:00:16.876531+0000","last_unstale":"2024-05-09T05:00:20.270704+0000","last_undegraded":"2024-05-09T05:00:20.270704+0000","last_fullsized":"2024-05-09T05:00:20.270704+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:54.996456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882435+0000","last_change":"2024-05-09T05:00:12.849957+0000","last_active":"2024-05-09T05:00:19.882435+0000","last_peered":"2024-05-09T05:00:19.882435+0000","last_clean":"2024-05-09T05:00:19.882435+0000","last_became_active":"2024-05-09T05:00:12.849138+0000","last_became_peered":"2024-05-09T05:00:12.849138+0000","last_unstale":"2024-05-09T05:00:19.882435+0000","last_undegraded":"2024-05-09T05:00:19.882435+0000","last_fullsized":"2024-05-09T05:00:19.882435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:02.740965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970665+0000","last_change":"2024-05-09T05:00:11.840734+0000","last_active":"2024-05-09T05:00:19.970665+0000","last_peered":"2024-05-09T05:00:19.970665+0000","last_clean":"2024-05-09T05:00:19.970665+0000","last_became_active":"2024-05-09T05:00:11.840463+0000","last_became_peered":"2024-05-09T05:00:11.840463+0000","last_unstale":"2024-05-09T05:00:19.970665+0000","last_undegraded":"2024-05-09T05:00:19.970665+0000","last_fullsized":"2024-05-09T05:00:19.970665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:49.660089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271060+0000","last_change":"2024-05-09T05:00:16.875120+0000","last_active":"2024-05-09T05:00:20.271060+0000","last_peered":"2024-05-09T05:00:20.271060+0000","last_clean":"2024-05-09T05:00:20.271060+0000","last_became_active":"2024-05-09T05:00:16.874728+0000","last_became_peered":"2024-05-09T05:00:16.874728+0000","last_unstale":"2024-05-09T05:00:20.271060+0000","last_undegraded":"2024-05-09T05:00:20.271060+0000","last_fullsized":"2024-05-09T05:00:20.271060+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:20.415639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271161+0000","last_change":"2024-05-09T05:00:12.851489+0000","last_active":"2024-05-09T05:00:20.271161+0000","last_peered":"2024-05-09T05:00:20.271161+0000","last_clean":"2024-05-09T05:00:20.271161+0000","last_became_active":"2024-05-09T05:00:12.841469+0000","last_became_peered":"2024-05-09T05:00:12.841469+0000","last_unstale":"2024-05-09T05:00:20.271161+0000","last_undegraded":"2024-05-09T05:00:20.271161+0000","last_fullsized":"2024-05-09T05:00:20.271161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:45.463031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860713+0000","last_change":"2024-05-09T05:00:11.841055+0000","last_active":"2024-05-09T05:00:17.860713+0000","last_peered":"2024-05-09T05:00:17.860713+0000","last_clean":"2024-05-09T05:00:17.860713+0000","last_became_active":"2024-05-09T05:00:11.839965+0000","last_became_peered":"2024-05-09T05:00:11.839965+0000","last_unstale":"2024-05-09T05:00:17.860713+0000","last_undegraded":"2024-05-09T05:00:17.860713+0000","last_fullsized":"2024-05-09T05:00:17.860713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:41:27.364045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271386+0000","last_change":"2024-05-09T05:00:16.873524+0000","last_active":"2024-05-09T05:00:20.271386+0000","last_peered":"2024-05-09T05:00:20.271386+0000","last_clean":"2024-05-09T05:00:20.271386+0000","last_became_active":"2024-05-09T05:00:16.872894+0000","last_became_peered":"2024-05-09T05:00:16.872894+0000","last_unstale":"2024-05-09T05:00:20.271386+0000","last_undegraded":"2024-05-09T05:00:20.271386+0000","last_fullsized":"2024-05-09T05:00:20.271386+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:32:00.425543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623506+0000","last_change":"2024-05-09T05:00:12.849280+0000","last_active":"2024-05-09T05:00:19.623506+0000","last_peered":"2024-05-09T05:00:19.623506+0000","last_clean":"2024-05-09T05:00:19.623506+0000","last_became_active":"2024-05-09T05:00:12.848968+0000","last_became_peered":"2024-05-09T05:00:12.848968+0000","last_unstale":"2024-05-09T05:00:19.623506+0000","last_undegraded":"2024-05-09T05:00:19.623506+0000","last_fullsized":"2024-05-09T05:00:19.623506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:49.330395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270849+0000","last_change":"2024-05-09T05:00:11.852903+0000","last_active":"2024-05-09T05:00:20.270849+0000","last_peered":"2024-05-09T05:00:20.270849+0000","last_clean":"2024-05-09T05:00:20.270849+0000","last_became_active":"2024-05-09T05:00:11.852584+0000","last_became_peered":"2024-05-09T05:00:11.852584+0000","last_unstale":"2024-05-09T05:00:20.270849+0000","last_undegraded":"2024-05-09T05:00:20.270849+0000","last_fullsized":"2024-05-09T05:00:20.270849+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:29:13.766278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624019+0000","last_change":"2024-05-09T05:00:16.882378+0000","last_active":"2024-05-09T05:00:19.624019+0000","last_peered":"2024-05-09T05:00:19.624019+0000","last_clean":"2024-05-09T05:00:19.624019+0000","last_became_active":"2024-05-09T05:00:16.881947+0000","last_became_peered":"2024-05-09T05:00:16.881947+0000","last_unstale":"2024-05-09T05:00:19.624019+0000","last_undegraded":"2024-05-09T05:00:19.624019+0000","last_fullsized":"2024-05-09T05:00:19.624019+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:32:00.425543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652370+0000","last_change":"2024-05-09T05:00:12.847632+0000","last_active":"2024-05-09T05:00:18.652370+0000","last_peered":"2024-05-09T05:00:18.652370+0000","last_clean":"2024-05-09T05:00:18.652370+0000","last_became_active":"2024-05-09T05:00:12.847358+0000","last_became_peered":"2024-05-09T05:00:12.847358+0000","last_unstale":"2024-05-09T05:00:18.652370+0000","last_undegraded":"2024-05-09T05:00:18.652370+0000","last_fullsized":"2024-05-09T05:00:18.652370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:49.330395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970011+0000","last_change":"2024-05-09T05:00:11.842747+0000","last_active":"2024-05-09T05:00:19.970011+0000","last_peered":"2024-05-09T05:00:19.970011+0000","last_clean":"2024-05-09T05:00:19.970011+0000","last_became_active":"2024-05-09T05:00:11.842474+0000","last_became_peered":"2024-05-09T05:00:11.842474+0000","last_unstale":"2024-05-09T05:00:19.970011+0000","last_undegraded":"2024-05-09T05:00:19.970011+0000","last_fullsized":"2024-05-09T05:00:19.970011+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:01.738771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":118,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271122+0000","last_change":"2024-05-09T05:00:08.667158+0000","last_active":"2024-05-09T05:00:20.271122+0000","last_peered":"2024-05-09T05:00:20.271122+0000","last_clean":"2024-05-09T05:00:20.271122+0000","last_became_active":"2024-05-09T05:00:08.666316+0000","last_became_peered":"2024-05-09T05:00:08.666316+0000","last_unstale":"2024-05-09T05:00:20.271122+0000","last_undegraded":"2024-05-09T05:00:20.271122+0000","last_fullsized":"2024-05-09T05:00:20.271122+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_clean_scrub_stamp":"2024-05-09T05:00:07.370119+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:20.234643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860739+0000","last_change":"2024-05-09T05:00:16.880367+0000","last_active":"2024-05-09T05:00:17.860739+0000","last_peered":"2024-05-09T05:00:17.860739+0000","last_clean":"2024-05-09T05:00:17.860739+0000","last_became_active":"2024-05-09T05:00:16.879980+0000","last_became_peered":"2024-05-09T05:00:16.879980+0000","last_unstale":"2024-05-09T05:00:17.860739+0000","last_undegraded":"2024-05-09T05:00:17.860739+0000","last_fullsized":"2024-05-09T05:00:17.860739+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:54.996456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271273+0000","last_change":"2024-05-09T05:00:12.852180+0000","last_active":"2024-05-09T05:00:20.271273+0000","last_peered":"2024-05-09T05:00:20.271273+0000","last_clean":"2024-05-09T05:00:20.271273+0000","last_became_active":"2024-05-09T05:00:12.851828+0000","last_became_peered":"2024-05-09T05:00:12.851828+0000","last_unstale":"2024-05-09T05:00:20.271273+0000","last_undegraded":"2024-05-09T05:00:20.271273+0000","last_fullsized":"2024-05-09T05:00:20.271273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:16.387563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.861065+0000","last_change":"2024-05-09T05:00:11.843024+0000","last_active":"2024-05-09T05:00:17.861065+0000","last_peered":"2024-05-09T05:00:17.861065+0000","last_clean":"2024-05-09T05:00:17.861065+0000","last_became_active":"2024-05-09T05:00:11.842703+0000","last_became_peered":"2024-05-09T05:00:11.842703+0000","last_unstale":"2024-05-09T05:00:17.861065+0000","last_undegraded":"2024-05-09T05:00:17.861065+0000","last_fullsized":"2024-05-09T05:00:17.861065+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:48:26.640184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860188+0000","last_change":"2024-05-09T05:00:16.879385+0000","last_active":"2024-05-09T05:00:17.860188+0000","last_peered":"2024-05-09T05:00:17.860188+0000","last_clean":"2024-05-09T05:00:17.860188+0000","last_became_active":"2024-05-09T05:00:16.878956+0000","last_became_peered":"2024-05-09T05:00:16.878956+0000","last_unstale":"2024-05-09T05:00:17.860188+0000","last_undegraded":"2024-05-09T05:00:17.860188+0000","last_fullsized":"2024-05-09T05:00:17.860188+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:49.342204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970802+0000","last_change":"2024-05-09T05:00:12.848110+0000","last_active":"2024-05-09T05:00:19.970802+0000","last_peered":"2024-05-09T05:00:19.970802+0000","last_clean":"2024-05-09T05:00:19.970802+0000","last_became_active":"2024-05-09T05:00:12.847628+0000","last_became_peered":"2024-05-09T05:00:12.847628+0000","last_unstale":"2024-05-09T05:00:19.970802+0000","last_undegraded":"2024-05-09T05:00:19.970802+0000","last_fullsized":"2024-05-09T05:00:19.970802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:16.387563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271019+0000","last_change":"2024-05-09T05:00:11.845441+0000","last_active":"2024-05-09T05:00:20.271019+0000","last_peered":"2024-05-09T05:00:20.271019+0000","last_clean":"2024-05-09T05:00:20.271019+0000","last_became_active":"2024-05-09T05:00:11.845151+0000","last_became_peered":"2024-05-09T05:00:11.845151+0000","last_unstale":"2024-05-09T05:00:20.271019+0000","last_undegraded":"2024-05-09T05:00:20.271019+0000","last_fullsized":"2024-05-09T05:00:20.271019+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:39.794811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623840+0000","last_change":"2024-05-09T05:00:16.880872+0000","last_active":"2024-05-09T05:00:19.623840+0000","last_peered":"2024-05-09T05:00:19.623840+0000","last_clean":"2024-05-09T05:00:19.623840+0000","last_became_active":"2024-05-09T05:00:16.876853+0000","last_became_peered":"2024-05-09T05:00:16.876853+0000","last_unstale":"2024-05-09T05:00:19.623840+0000","last_undegraded":"2024-05-09T05:00:19.623840+0000","last_fullsized":"2024-05-09T05:00:19.623840+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:47:02.206708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271556+0000","last_change":"2024-05-09T05:00:12.854694+0000","last_active":"2024-05-09T05:00:20.271556+0000","last_peered":"2024-05-09T05:00:20.271556+0000","last_clean":"2024-05-09T05:00:20.271556+0000","last_became_active":"2024-05-09T05:00:12.854098+0000","last_became_peered":"2024-05-09T05:00:12.854098+0000","last_unstale":"2024-05-09T05:00:20.271556+0000","last_undegraded":"2024-05-09T05:00:20.271556+0000","last_fullsized":"2024-05-09T05:00:20.271556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:23.343124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860998+0000","last_change":"2024-05-09T05:00:11.835134+0000","last_active":"2024-05-09T05:00:17.860998+0000","last_peered":"2024-05-09T05:00:17.860998+0000","last_clean":"2024-05-09T05:00:17.860998+0000","last_became_active":"2024-05-09T05:00:11.834773+0000","last_became_peered":"2024-05-09T05:00:11.834773+0000","last_unstale":"2024-05-09T05:00:17.860998+0000","last_undegraded":"2024-05-09T05:00:17.860998+0000","last_fullsized":"2024-05-09T05:00:17.860998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:50.824636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970193+0000","last_change":"2024-05-09T05:00:16.879633+0000","last_active":"2024-05-09T05:00:19.970193+0000","last_peered":"2024-05-09T05:00:19.970193+0000","last_clean":"2024-05-09T05:00:19.970193+0000","last_became_active":"2024-05-09T05:00:16.879232+0000","last_became_peered":"2024-05-09T05:00:16.879232+0000","last_unstale":"2024-05-09T05:00:19.970193+0000","last_undegraded":"2024-05-09T05:00:19.970193+0000","last_fullsized":"2024-05-09T05:00:19.970193+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:18:15.413800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271558+0000","last_change":"2024-05-09T05:00:12.852679+0000","last_active":"2024-05-09T05:00:20.271558+0000","last_peered":"2024-05-09T05:00:20.271558+0000","last_clean":"2024-05-09T05:00:20.271558+0000","last_became_active":"2024-05-09T05:00:12.852434+0000","last_became_peered":"2024-05-09T05:00:12.852434+0000","last_unstale":"2024-05-09T05:00:20.271558+0000","last_undegraded":"2024-05-09T05:00:20.271558+0000","last_fullsized":"2024-05-09T05:00:20.271558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:06.793812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881482+0000","last_change":"2024-05-09T05:00:11.845447+0000","last_active":"2024-05-09T05:00:19.881482+0000","last_peered":"2024-05-09T05:00:19.881482+0000","last_clean":"2024-05-09T05:00:19.881482+0000","last_became_active":"2024-05-09T05:00:11.845139+0000","last_became_peered":"2024-05-09T05:00:11.845139+0000","last_unstale":"2024-05-09T05:00:19.881482+0000","last_undegraded":"2024-05-09T05:00:19.881482+0000","last_fullsized":"2024-05-09T05:00:19.881482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:32:22.225189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881692+0000","last_change":"2024-05-09T05:00:16.877859+0000","last_active":"2024-05-09T05:00:19.881692+0000","last_peered":"2024-05-09T05:00:19.881692+0000","last_clean":"2024-05-09T05:00:19.881692+0000","last_became_active":"2024-05-09T05:00:16.877418+0000","last_became_peered":"2024-05-09T05:00:16.877418+0000","last_unstale":"2024-05-09T05:00:19.881692+0000","last_undegraded":"2024-05-09T05:00:19.881692+0000","last_fullsized":"2024-05-09T05:00:19.881692+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:28.924332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271116+0000","last_change":"2024-05-09T05:00:12.855313+0000","last_active":"2024-05-09T05:00:20.271116+0000","last_peered":"2024-05-09T05:00:20.271116+0000","last_clean":"2024-05-09T05:00:20.271116+0000","last_became_active":"2024-05-09T05:00:12.855041+0000","last_became_peered":"2024-05-09T05:00:12.855041+0000","last_unstale":"2024-05-09T05:00:20.271116+0000","last_undegraded":"2024-05-09T05:00:20.271116+0000","last_fullsized":"2024-05-09T05:00:20.271116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:01:08.488591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860579+0000","last_change":"2024-05-09T05:00:11.839443+0000","last_active":"2024-05-09T05:00:17.860579+0000","last_peered":"2024-05-09T05:00:17.860579+0000","last_clean":"2024-05-09T05:00:17.860579+0000","last_became_active":"2024-05-09T05:00:11.839169+0000","last_became_peered":"2024-05-09T05:00:11.839169+0000","last_unstale":"2024-05-09T05:00:17.860579+0000","last_undegraded":"2024-05-09T05:00:17.860579+0000","last_fullsized":"2024-05-09T05:00:17.860579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:10.779395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623875+0000","last_change":"2024-05-09T05:00:16.876207+0000","last_active":"2024-05-09T05:00:19.623875+0000","last_peered":"2024-05-09T05:00:19.623875+0000","last_clean":"2024-05-09T05:00:19.623875+0000","last_became_active":"2024-05-09T05:00:16.875739+0000","last_became_peered":"2024-05-09T05:00:16.875739+0000","last_unstale":"2024-05-09T05:00:19.623875+0000","last_undegraded":"2024-05-09T05:00:19.623875+0000","last_fullsized":"2024-05-09T05:00:19.623875+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:10.821888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860230+0000","last_change":"2024-05-09T05:00:12.851872+0000","last_active":"2024-05-09T05:00:17.860230+0000","last_peered":"2024-05-09T05:00:17.860230+0000","last_clean":"2024-05-09T05:00:17.860230+0000","last_became_active":"2024-05-09T05:00:12.851590+0000","last_became_peered":"2024-05-09T05:00:12.851590+0000","last_unstale":"2024-05-09T05:00:17.860230+0000","last_undegraded":"2024-05-09T05:00:17.860230+0000","last_fullsized":"2024-05-09T05:00:17.860230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:49.330395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860330+0000","last_change":"2024-05-09T05:00:11.845978+0000","last_active":"2024-05-09T05:00:17.860330+0000","last_peered":"2024-05-09T05:00:17.860330+0000","last_clean":"2024-05-09T05:00:17.860330+0000","last_became_active":"2024-05-09T05:00:11.845129+0000","last_became_peered":"2024-05-09T05:00:11.845129+0000","last_unstale":"2024-05-09T05:00:17.860330+0000","last_undegraded":"2024-05-09T05:00:17.860330+0000","last_fullsized":"2024-05-09T05:00:17.860330+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:49.660089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652710+0000","last_change":"2024-05-09T05:00:16.882756+0000","last_active":"2024-05-09T05:00:18.652710+0000","last_peered":"2024-05-09T05:00:18.652710+0000","last_clean":"2024-05-09T05:00:18.652710+0000","last_became_active":"2024-05-09T05:00:16.882378+0000","last_became_peered":"2024-05-09T05:00:16.882378+0000","last_unstale":"2024-05-09T05:00:18.652710+0000","last_undegraded":"2024-05-09T05:00:18.652710+0000","last_fullsized":"2024-05-09T05:00:18.652710+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:51:44.393878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271158+0000","last_change":"2024-05-09T05:00:12.854634+0000","last_active":"2024-05-09T05:00:20.271158+0000","last_peered":"2024-05-09T05:00:20.271158+0000","last_clean":"2024-05-09T05:00:20.271158+0000","last_became_active":"2024-05-09T05:00:12.854366+0000","last_became_peered":"2024-05-09T05:00:12.854366+0000","last_unstale":"2024-05-09T05:00:20.271158+0000","last_undegraded":"2024-05-09T05:00:20.271158+0000","last_fullsized":"2024-05-09T05:00:20.271158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:45.314128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.969990+0000","last_change":"2024-05-09T05:00:11.843045+0000","last_active":"2024-05-09T05:00:19.969990+0000","last_peered":"2024-05-09T05:00:19.969990+0000","last_clean":"2024-05-09T05:00:19.969990+0000","last_became_active":"2024-05-09T05:00:11.842728+0000","last_became_peered":"2024-05-09T05:00:11.842728+0000","last_unstale":"2024-05-09T05:00:19.969990+0000","last_undegraded":"2024-05-09T05:00:19.969990+0000","last_fullsized":"2024-05-09T05:00:19.969990+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:25.803010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652628+0000","last_change":"2024-05-09T05:00:16.880672+0000","last_active":"2024-05-09T05:00:18.652628+0000","last_peered":"2024-05-09T05:00:18.652628+0000","last_clean":"2024-05-09T05:00:18.652628+0000","last_became_active":"2024-05-09T05:00:16.880274+0000","last_became_peered":"2024-05-09T05:00:16.880274+0000","last_unstale":"2024-05-09T05:00:18.652628+0000","last_undegraded":"2024-05-09T05:00:18.652628+0000","last_fullsized":"2024-05-09T05:00:18.652628+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:27:18.295238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623650+0000","last_change":"2024-05-09T05:00:12.854427+0000","last_active":"2024-05-09T05:00:19.623650+0000","last_peered":"2024-05-09T05:00:19.623650+0000","last_clean":"2024-05-09T05:00:19.623650+0000","last_became_active":"2024-05-09T05:00:12.854089+0000","last_became_peered":"2024-05-09T05:00:12.854089+0000","last_unstale":"2024-05-09T05:00:19.623650+0000","last_undegraded":"2024-05-09T05:00:19.623650+0000","last_fullsized":"2024-05-09T05:00:19.623650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:33:12.899897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623926+0000","last_change":"2024-05-09T05:00:11.845606+0000","last_active":"2024-05-09T05:00:19.623926+0000","last_peered":"2024-05-09T05:00:19.623926+0000","last_clean":"2024-05-09T05:00:19.623926+0000","last_became_active":"2024-05-09T05:00:11.845227+0000","last_became_peered":"2024-05-09T05:00:11.845227+0000","last_unstale":"2024-05-09T05:00:19.623926+0000","last_undegraded":"2024-05-09T05:00:19.623926+0000","last_fullsized":"2024-05-09T05:00:19.623926+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:50.824636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970860+0000","last_change":"2024-05-09T05:00:16.880664+0000","last_active":"2024-05-09T05:00:19.970860+0000","last_peered":"2024-05-09T05:00:19.970860+0000","last_clean":"2024-05-09T05:00:19.970860+0000","last_became_active":"2024-05-09T05:00:16.880226+0000","last_became_peered":"2024-05-09T05:00:16.880226+0000","last_unstale":"2024-05-09T05:00:19.970860+0000","last_undegraded":"2024-05-09T05:00:19.970860+0000","last_fullsized":"2024-05-09T05:00:19.970860+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:41.016707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271152+0000","last_change":"2024-05-09T05:00:12.858289+0000","last_active":"2024-05-09T05:00:20.271152+0000","last_peered":"2024-05-09T05:00:20.271152+0000","last_clean":"2024-05-09T05:00:20.271152+0000","last_became_active":"2024-05-09T05:00:12.858013+0000","last_became_peered":"2024-05-09T05:00:12.858013+0000","last_unstale":"2024-05-09T05:00:20.271152+0000","last_undegraded":"2024-05-09T05:00:20.271152+0000","last_fullsized":"2024-05-09T05:00:20.271152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:50.968380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:22.574608+0000","last_change":"2024-05-09T05:00:11.842419+0000","last_active":"2024-05-09T05:00:22.574608+0000","last_peered":"2024-05-09T05:00:22.574608+0000","last_clean":"2024-05-09T05:00:22.574608+0000","last_became_active":"2024-05-09T05:00:11.842112+0000","last_became_peered":"2024-05-09T05:00:11.842112+0000","last_unstale":"2024-05-09T05:00:22.574608+0000","last_undegraded":"2024-05-09T05:00:22.574608+0000","last_fullsized":"2024-05-09T05:00:22.574608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:25.803010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271066+0000","last_change":"2024-05-09T05:00:16.874021+0000","last_active":"2024-05-09T05:00:20.271066+0000","last_peered":"2024-05-09T05:00:20.271066+0000","last_clean":"2024-05-09T05:00:20.271066+0000","last_became_active":"2024-05-09T05:00:16.873587+0000","last_became_peered":"2024-05-09T05:00:16.873587+0000","last_unstale":"2024-05-09T05:00:20.271066+0000","last_undegraded":"2024-05-09T05:00:20.271066+0000","last_fullsized":"2024-05-09T05:00:20.271066+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:47:02.206708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882366+0000","last_change":"2024-05-09T05:00:12.844150+0000","last_active":"2024-05-09T05:00:19.882366+0000","last_peered":"2024-05-09T05:00:19.882366+0000","last_clean":"2024-05-09T05:00:19.882366+0000","last_became_active":"2024-05-09T05:00:12.843845+0000","last_became_peered":"2024-05-09T05:00:12.843845+0000","last_unstale":"2024-05-09T05:00:19.882366+0000","last_undegraded":"2024-05-09T05:00:19.882366+0000","last_fullsized":"2024-05-09T05:00:19.882366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:27:58.422255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270961+0000","last_change":"2024-05-09T05:00:11.842387+0000","last_active":"2024-05-09T05:00:20.270961+0000","last_peered":"2024-05-09T05:00:20.270961+0000","last_clean":"2024-05-09T05:00:20.270961+0000","last_became_active":"2024-05-09T05:00:11.840749+0000","last_became_peered":"2024-05-09T05:00:11.840749+0000","last_unstale":"2024-05-09T05:00:20.270961+0000","last_undegraded":"2024-05-09T05:00:20.270961+0000","last_fullsized":"2024-05-09T05:00:20.270961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:27:57.420062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882161+0000","last_change":"2024-05-09T05:00:16.877382+0000","last_active":"2024-05-09T05:00:19.882161+0000","last_peered":"2024-05-09T05:00:19.882161+0000","last_clean":"2024-05-09T05:00:19.882161+0000","last_became_active":"2024-05-09T05:00:16.876966+0000","last_became_peered":"2024-05-09T05:00:16.876966+0000","last_unstale":"2024-05-09T05:00:19.882161+0000","last_undegraded":"2024-05-09T05:00:19.882161+0000","last_fullsized":"2024-05-09T05:00:19.882161+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:20:41.130506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624095+0000","last_change":"2024-05-09T05:00:12.844211+0000","last_active":"2024-05-09T05:00:19.624095+0000","last_peered":"2024-05-09T05:00:19.624095+0000","last_clean":"2024-05-09T05:00:19.624095+0000","last_became_active":"2024-05-09T05:00:12.843545+0000","last_became_peered":"2024-05-09T05:00:12.843545+0000","last_unstale":"2024-05-09T05:00:19.624095+0000","last_undegraded":"2024-05-09T05:00:19.624095+0000","last_fullsized":"2024-05-09T05:00:19.624095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970032+0000","last_change":"2024-05-09T05:00:11.841156+0000","last_active":"2024-05-09T05:00:19.970032+0000","last_peered":"2024-05-09T05:00:19.970032+0000","last_clean":"2024-05-09T05:00:19.970032+0000","last_became_active":"2024-05-09T05:00:11.840620+0000","last_became_peered":"2024-05-09T05:00:11.840620+0000","last_unstale":"2024-05-09T05:00:19.970032+0000","last_undegraded":"2024-05-09T05:00:19.970032+0000","last_fullsized":"2024-05-09T05:00:19.970032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:10.779395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652218+0000","last_change":"2024-05-09T05:00:16.875531+0000","last_active":"2024-05-09T05:00:18.652218+0000","last_peered":"2024-05-09T05:00:18.652218+0000","last_clean":"2024-05-09T05:00:18.652218+0000","last_became_active":"2024-05-09T05:00:16.875056+0000","last_became_peered":"2024-05-09T05:00:16.875056+0000","last_unstale":"2024-05-09T05:00:18.652218+0000","last_undegraded":"2024-05-09T05:00:18.652218+0000","last_fullsized":"2024-05-09T05:00:18.652218+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:41.016707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970758+0000","last_change":"2024-05-09T05:00:12.852295+0000","last_active":"2024-05-09T05:00:19.970758+0000","last_peered":"2024-05-09T05:00:19.970758+0000","last_clean":"2024-05-09T05:00:19.970758+0000","last_became_active":"2024-05-09T05:00:12.851993+0000","last_became_peered":"2024-05-09T05:00:12.851993+0000","last_unstale":"2024-05-09T05:00:19.970758+0000","last_undegraded":"2024-05-09T05:00:19.970758+0000","last_fullsized":"2024-05-09T05:00:19.970758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:44.359533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860982+0000","last_change":"2024-05-09T05:00:11.840063+0000","last_active":"2024-05-09T05:00:17.860982+0000","last_peered":"2024-05-09T05:00:17.860982+0000","last_clean":"2024-05-09T05:00:17.860982+0000","last_became_active":"2024-05-09T05:00:11.839772+0000","last_became_peered":"2024-05-09T05:00:11.839772+0000","last_unstale":"2024-05-09T05:00:17.860982+0000","last_undegraded":"2024-05-09T05:00:17.860982+0000","last_fullsized":"2024-05-09T05:00:17.860982+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:01.738771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652211+0000","last_change":"2024-05-09T05:00:16.883403+0000","last_active":"2024-05-09T05:00:18.652211+0000","last_peered":"2024-05-09T05:00:18.652211+0000","last_clean":"2024-05-09T05:00:18.652211+0000","last_became_active":"2024-05-09T05:00:16.882440+0000","last_became_peered":"2024-05-09T05:00:16.882440+0000","last_unstale":"2024-05-09T05:00:18.652211+0000","last_undegraded":"2024-05-09T05:00:18.652211+0000","last_fullsized":"2024-05-09T05:00:18.652211+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:18:15.413800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624058+0000","last_change":"2024-05-09T05:00:16.883006+0000","last_active":"2024-05-09T05:00:19.624058+0000","last_peered":"2024-05-09T05:00:19.624058+0000","last_clean":"2024-05-09T05:00:19.624058+0000","last_became_active":"2024-05-09T05:00:16.882624+0000","last_became_peered":"2024-05-09T05:00:16.882624+0000","last_unstale":"2024-05-09T05:00:19.624058+0000","last_undegraded":"2024-05-09T05:00:19.624058+0000","last_fullsized":"2024-05-09T05:00:19.624058+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:02.171370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270485+0000","last_change":"2024-05-09T05:00:12.850411+0000","last_active":"2024-05-09T05:00:20.270485+0000","last_peered":"2024-05-09T05:00:20.270485+0000","last_clean":"2024-05-09T05:00:20.270485+0000","last_became_active":"2024-05-09T05:00:12.850121+0000","last_became_peered":"2024-05-09T05:00:12.850121+0000","last_unstale":"2024-05-09T05:00:20.270485+0000","last_undegraded":"2024-05-09T05:00:20.270485+0000","last_fullsized":"2024-05-09T05:00:20.270485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:30.691332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970418+0000","last_change":"2024-05-09T05:00:11.843746+0000","last_active":"2024-05-09T05:00:19.970418+0000","last_peered":"2024-05-09T05:00:19.970418+0000","last_clean":"2024-05-09T05:00:19.970418+0000","last_became_active":"2024-05-09T05:00:11.843423+0000","last_became_peered":"2024-05-09T05:00:11.843423+0000","last_unstale":"2024-05-09T05:00:19.970418+0000","last_undegraded":"2024-05-09T05:00:19.970418+0000","last_fullsized":"2024-05-09T05:00:19.970418+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:41:27.364045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860975+0000","last_change":"2024-05-09T05:00:16.873412+0000","last_active":"2024-05-09T05:00:17.860975+0000","last_peered":"2024-05-09T05:00:17.860975+0000","last_clean":"2024-05-09T05:00:17.860975+0000","last_became_active":"2024-05-09T05:00:16.873014+0000","last_became_peered":"2024-05-09T05:00:16.873014+0000","last_unstale":"2024-05-09T05:00:17.860975+0000","last_undegraded":"2024-05-09T05:00:17.860975+0000","last_fullsized":"2024-05-09T05:00:17.860975+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:20.415639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652208+0000","last_change":"2024-05-09T05:00:12.856243+0000","last_active":"2024-05-09T05:00:18.652208+0000","last_peered":"2024-05-09T05:00:18.652208+0000","last_clean":"2024-05-09T05:00:18.652208+0000","last_became_active":"2024-05-09T05:00:12.854306+0000","last_became_peered":"2024-05-09T05:00:12.854306+0000","last_unstale":"2024-05-09T05:00:18.652208+0000","last_undegraded":"2024-05-09T05:00:18.652208+0000","last_fullsized":"2024-05-09T05:00:18.652208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:10:10.149613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652448+0000","last_change":"2024-05-09T05:00:11.837046+0000","last_active":"2024-05-09T05:00:18.652448+0000","last_peered":"2024-05-09T05:00:18.652448+0000","last_clean":"2024-05-09T05:00:18.652448+0000","last_became_active":"2024-05-09T05:00:11.836773+0000","last_became_peered":"2024-05-09T05:00:11.836773+0000","last_unstale":"2024-05-09T05:00:18.652448+0000","last_undegraded":"2024-05-09T05:00:18.652448+0000","last_fullsized":"2024-05-09T05:00:18.652448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:48:26.640184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860385+0000","last_change":"2024-05-09T05:00:11.841442+0000","last_active":"2024-05-09T05:00:17.860385+0000","last_peered":"2024-05-09T05:00:17.860385+0000","last_clean":"2024-05-09T05:00:17.860385+0000","last_became_active":"2024-05-09T05:00:11.841099+0000","last_became_peered":"2024-05-09T05:00:11.841099+0000","last_unstale":"2024-05-09T05:00:17.860385+0000","last_undegraded":"2024-05-09T05:00:17.860385+0000","last_fullsized":"2024-05-09T05:00:17.860385+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:41:27.364045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623878+0000","last_change":"2024-05-09T05:00:12.845049+0000","last_active":"2024-05-09T05:00:19.623878+0000","last_peered":"2024-05-09T05:00:19.623878+0000","last_clean":"2024-05-09T05:00:19.623878+0000","last_became_active":"2024-05-09T05:00:12.844724+0000","last_became_peered":"2024-05-09T05:00:12.844724+0000","last_unstale":"2024-05-09T05:00:19.623878+0000","last_undegraded":"2024-05-09T05:00:19.623878+0000","last_fullsized":"2024-05-09T05:00:19.623878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:45.463031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860933+0000","last_change":"2024-05-09T05:00:16.883497+0000","last_active":"2024-05-09T05:00:17.860933+0000","last_peered":"2024-05-09T05:00:17.860933+0000","last_clean":"2024-05-09T05:00:17.860933+0000","last_became_active":"2024-05-09T05:00:16.883063+0000","last_became_peered":"2024-05-09T05:00:16.883063+0000","last_unstale":"2024-05-09T05:00:17.860933+0000","last_undegraded":"2024-05-09T05:00:17.860933+0000","last_fullsized":"2024-05-09T05:00:17.860933+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:04:58.832049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652723+0000","last_change":"2024-05-09T05:00:11.847505+0000","last_active":"2024-05-09T05:00:18.652723+0000","last_peered":"2024-05-09T05:00:18.652723+0000","last_clean":"2024-05-09T05:00:18.652723+0000","last_became_active":"2024-05-09T05:00:11.847187+0000","last_became_peered":"2024-05-09T05:00:11.847187+0000","last_unstale":"2024-05-09T05:00:18.652723+0000","last_undegraded":"2024-05-09T05:00:18.652723+0000","last_fullsized":"2024-05-09T05:00:18.652723+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:50.824636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882122+0000","last_change":"2024-05-09T05:00:12.838998+0000","last_active":"2024-05-09T05:00:19.882122+0000","last_peered":"2024-05-09T05:00:19.882122+0000","last_clean":"2024-05-09T05:00:19.882122+0000","last_became_active":"2024-05-09T05:00:12.838702+0000","last_became_peered":"2024-05-09T05:00:12.838702+0000","last_unstale":"2024-05-09T05:00:19.882122+0000","last_undegraded":"2024-05-09T05:00:19.882122+0000","last_fullsized":"2024-05-09T05:00:19.882122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:50.662283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652939+0000","last_change":"2024-05-09T05:00:16.876805+0000","last_active":"2024-05-09T05:00:18.652939+0000","last_peered":"2024-05-09T05:00:18.652939+0000","last_clean":"2024-05-09T05:00:18.652939+0000","last_became_active":"2024-05-09T05:00:16.876265+0000","last_became_peered":"2024-05-09T05:00:16.876265+0000","last_unstale":"2024-05-09T05:00:18.652939+0000","last_undegraded":"2024-05-09T05:00:18.652939+0000","last_fullsized":"2024-05-09T05:00:18.652939+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:54:27.622037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860099+0000","last_change":"2024-05-09T05:00:12.857527+0000","last_active":"2024-05-09T05:00:17.860099+0000","last_peered":"2024-05-09T05:00:17.860099+0000","last_clean":"2024-05-09T05:00:17.860099+0000","last_became_active":"2024-05-09T05:00:12.857202+0000","last_became_peered":"2024-05-09T05:00:12.857202+0000","last_unstale":"2024-05-09T05:00:17.860099+0000","last_undegraded":"2024-05-09T05:00:17.860099+0000","last_fullsized":"2024-05-09T05:00:17.860099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:45.463031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860631+0000","last_change":"2024-05-09T05:00:12.844334+0000","last_active":"2024-05-09T05:00:17.860631+0000","last_peered":"2024-05-09T05:00:17.860631+0000","last_clean":"2024-05-09T05:00:17.860631+0000","last_became_active":"2024-05-09T05:00:12.844018+0000","last_became_peered":"2024-05-09T05:00:12.844018+0000","last_unstale":"2024-05-09T05:00:17.860631+0000","last_undegraded":"2024-05-09T05:00:17.860631+0000","last_fullsized":"2024-05-09T05:00:17.860631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:40.797004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970591+0000","last_change":"2024-05-09T05:00:12.855985+0000","last_active":"2024-05-09T05:00:19.970591+0000","last_peered":"2024-05-09T05:00:19.970591+0000","last_clean":"2024-05-09T05:00:19.970591+0000","last_became_active":"2024-05-09T05:00:12.855586+0000","last_became_peered":"2024-05-09T05:00:12.855586+0000","last_unstale":"2024-05-09T05:00:19.970591+0000","last_undegraded":"2024-05-09T05:00:19.970591+0000","last_fullsized":"2024-05-09T05:00:19.970591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:06.793812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623781+0000","last_change":"2024-05-09T05:00:12.848323+0000","last_active":"2024-05-09T05:00:19.623781+0000","last_peered":"2024-05-09T05:00:19.623781+0000","last_clean":"2024-05-09T05:00:19.623781+0000","last_became_active":"2024-05-09T05:00:12.848053+0000","last_became_peered":"2024-05-09T05:00:12.848053+0000","last_unstale":"2024-05-09T05:00:19.623781+0000","last_undegraded":"2024-05-09T05:00:19.623781+0000","last_fullsized":"2024-05-09T05:00:19.623781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:19.080163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652749+0000","last_change":"2024-05-09T05:00:12.849804+0000","last_active":"2024-05-09T05:00:18.652749+0000","last_peered":"2024-05-09T05:00:18.652749+0000","last_clean":"2024-05-09T05:00:18.652749+0000","last_became_active":"2024-05-09T05:00:12.849530+0000","last_became_peered":"2024-05-09T05:00:12.849530+0000","last_unstale":"2024-05-09T05:00:18.652749+0000","last_undegraded":"2024-05-09T05:00:18.652749+0000","last_fullsized":"2024-05-09T05:00:18.652749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:20:37.102430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270858+0000","last_change":"2024-05-09T05:00:12.848438+0000","last_active":"2024-05-09T05:00:20.270858+0000","last_peered":"2024-05-09T05:00:20.270858+0000","last_clean":"2024-05-09T05:00:20.270858+0000","last_became_active":"2024-05-09T05:00:12.848176+0000","last_became_peered":"2024-05-09T05:00:12.848176+0000","last_unstale":"2024-05-09T05:00:20.270858+0000","last_undegraded":"2024-05-09T05:00:20.270858+0000","last_fullsized":"2024-05-09T05:00:20.270858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860997+0000","last_change":"2024-05-09T05:00:12.855310+0000","last_active":"2024-05-09T05:00:17.860997+0000","last_peered":"2024-05-09T05:00:17.860997+0000","last_clean":"2024-05-09T05:00:17.860997+0000","last_became_active":"2024-05-09T05:00:12.854031+0000","last_became_peered":"2024-05-09T05:00:12.854031+0000","last_unstale":"2024-05-09T05:00:17.860997+0000","last_undegraded":"2024-05-09T05:00:17.860997+0000","last_fullsized":"2024-05-09T05:00:17.860997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:44.359533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652800+0000","last_change":"2024-05-09T05:00:12.849125+0000","last_active":"2024-05-09T05:00:18.652800+0000","last_peered":"2024-05-09T05:00:18.652800+0000","last_clean":"2024-05-09T05:00:18.652800+0000","last_became_active":"2024-05-09T05:00:12.848852+0000","last_became_peered":"2024-05-09T05:00:12.848852+0000","last_unstale":"2024-05-09T05:00:18.652800+0000","last_undegraded":"2024-05-09T05:00:18.652800+0000","last_fullsized":"2024-05-09T05:00:18.652800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970301+0000","last_change":"2024-05-09T05:00:12.847892+0000","last_active":"2024-05-09T05:00:19.970301+0000","last_peered":"2024-05-09T05:00:19.970301+0000","last_clean":"2024-05-09T05:00:19.970301+0000","last_became_active":"2024-05-09T05:00:12.847051+0000","last_became_peered":"2024-05-09T05:00:12.847051+0000","last_unstale":"2024-05-09T05:00:19.970301+0000","last_undegraded":"2024-05-09T05:00:19.970301+0000","last_fullsized":"2024-05-09T05:00:19.970301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652323+0000","last_change":"2024-05-09T05:00:12.848463+0000","last_active":"2024-05-09T05:00:18.652323+0000","last_peered":"2024-05-09T05:00:18.652323+0000","last_clean":"2024-05-09T05:00:18.652323+0000","last_became_active":"2024-05-09T05:00:12.848170+0000","last_became_peered":"2024-05-09T05:00:12.848170+0000","last_unstale":"2024-05-09T05:00:18.652323+0000","last_undegraded":"2024-05-09T05:00:18.652323+0000","last_fullsized":"2024-05-09T05:00:18.652323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:45.463031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970492+0000","last_change":"2024-05-09T05:00:12.852750+0000","last_active":"2024-05-09T05:00:19.970492+0000","last_peered":"2024-05-09T05:00:19.970492+0000","last_clean":"2024-05-09T05:00:19.970492+0000","last_became_active":"2024-05-09T05:00:12.852487+0000","last_became_peered":"2024-05-09T05:00:12.852487+0000","last_unstale":"2024-05-09T05:00:19.970492+0000","last_undegraded":"2024-05-09T05:00:19.970492+0000","last_fullsized":"2024-05-09T05:00:19.970492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:50.968380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623652+0000","last_change":"2024-05-09T05:00:12.850281+0000","last_active":"2024-05-09T05:00:19.623652+0000","last_peered":"2024-05-09T05:00:19.623652+0000","last_clean":"2024-05-09T05:00:19.623652+0000","last_became_active":"2024-05-09T05:00:12.849875+0000","last_became_peered":"2024-05-09T05:00:12.849875+0000","last_unstale":"2024-05-09T05:00:19.623652+0000","last_undegraded":"2024-05-09T05:00:19.623652+0000","last_fullsized":"2024-05-09T05:00:19.623652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:10:10.149613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860536+0000","last_change":"2024-05-09T05:00:12.841388+0000","last_active":"2024-05-09T05:00:17.860536+0000","last_peered":"2024-05-09T05:00:17.860536+0000","last_clean":"2024-05-09T05:00:17.860536+0000","last_became_active":"2024-05-09T05:00:12.841121+0000","last_became_peered":"2024-05-09T05:00:12.841121+0000","last_unstale":"2024-05-09T05:00:17.860536+0000","last_undegraded":"2024-05-09T05:00:17.860536+0000","last_fullsized":"2024-05-09T05:00:17.860536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:30.691332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271354+0000","last_change":"2024-05-09T05:00:12.853880+0000","last_active":"2024-05-09T05:00:20.271354+0000","last_peered":"2024-05-09T05:00:20.271354+0000","last_clean":"2024-05-09T05:00:20.271354+0000","last_became_active":"2024-05-09T05:00:12.852044+0000","last_became_peered":"2024-05-09T05:00:12.852044+0000","last_unstale":"2024-05-09T05:00:20.271354+0000","last_undegraded":"2024-05-09T05:00:20.271354+0000","last_fullsized":"2024-05-09T05:00:20.271354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:31:56.397467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860942+0000","last_change":"2024-05-09T05:00:12.842477+0000","last_active":"2024-05-09T05:00:17.860942+0000","last_peered":"2024-05-09T05:00:17.860942+0000","last_clean":"2024-05-09T05:00:17.860942+0000","last_became_active":"2024-05-09T05:00:12.842135+0000","last_became_peered":"2024-05-09T05:00:12.842135+0000","last_unstale":"2024-05-09T05:00:17.860942+0000","last_undegraded":"2024-05-09T05:00:17.860942+0000","last_fullsized":"2024-05-09T05:00:17.860942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:27:58.422255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.971029+0000","last_change":"2024-05-09T05:00:12.852559+0000","last_active":"2024-05-09T05:00:19.971029+0000","last_peered":"2024-05-09T05:00:19.971029+0000","last_clean":"2024-05-09T05:00:19.971029+0000","last_became_active":"2024-05-09T05:00:12.852370+0000","last_became_peered":"2024-05-09T05:00:12.852370+0000","last_unstale":"2024-05-09T05:00:19.971029+0000","last_undegraded":"2024-05-09T05:00:19.971029+0000","last_fullsized":"2024-05-09T05:00:19.971029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:58.178632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970409+0000","last_change":"2024-05-09T05:00:12.856404+0000","last_active":"2024-05-09T05:00:19.970409+0000","last_peered":"2024-05-09T05:00:19.970409+0000","last_clean":"2024-05-09T05:00:19.970409+0000","last_became_active":"2024-05-09T05:00:12.856126+0000","last_became_peered":"2024-05-09T05:00:12.856126+0000","last_unstale":"2024-05-09T05:00:19.970409+0000","last_undegraded":"2024-05-09T05:00:19.970409+0000","last_fullsized":"2024-05-09T05:00:19.970409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270574+0000","last_change":"2024-05-09T05:00:12.847697+0000","last_active":"2024-05-09T05:00:20.270574+0000","last_peered":"2024-05-09T05:00:20.270574+0000","last_clean":"2024-05-09T05:00:20.270574+0000","last_became_active":"2024-05-09T05:00:12.847412+0000","last_became_peered":"2024-05-09T05:00:12.847412+0000","last_unstale":"2024-05-09T05:00:20.270574+0000","last_undegraded":"2024-05-09T05:00:20.270574+0000","last_fullsized":"2024-05-09T05:00:20.270574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:19.080163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623651+0000","last_change":"2024-05-09T05:00:12.846578+0000","last_active":"2024-05-09T05:00:19.623651+0000","last_peered":"2024-05-09T05:00:19.623651+0000","last_clean":"2024-05-09T05:00:19.623651+0000","last_became_active":"2024-05-09T05:00:12.844465+0000","last_became_peered":"2024-05-09T05:00:12.844465+0000","last_unstale":"2024-05-09T05:00:19.623651+0000","last_undegraded":"2024-05-09T05:00:19.623651+0000","last_fullsized":"2024-05-09T05:00:19.623651+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:17:11.682946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652503+0000","last_change":"2024-05-09T05:00:12.852179+0000","last_active":"2024-05-09T05:00:18.652503+0000","last_peered":"2024-05-09T05:00:18.652503+0000","last_clean":"2024-05-09T05:00:18.652503+0000","last_became_active":"2024-05-09T05:00:12.849715+0000","last_became_peered":"2024-05-09T05:00:12.849715+0000","last_unstale":"2024-05-09T05:00:18.652503+0000","last_undegraded":"2024-05-09T05:00:18.652503+0000","last_fullsized":"2024-05-09T05:00:18.652503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:19.080163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882233+0000","last_change":"2024-05-09T05:00:12.837696+0000","last_active":"2024-05-09T05:00:19.882233+0000","last_peered":"2024-05-09T05:00:19.882233+0000","last_clean":"2024-05-09T05:00:19.882233+0000","last_became_active":"2024-05-09T05:00:12.837445+0000","last_became_peered":"2024-05-09T05:00:12.837445+0000","last_unstale":"2024-05-09T05:00:19.882233+0000","last_undegraded":"2024-05-09T05:00:19.882233+0000","last_fullsized":"2024-05-09T05:00:19.882233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:40.797004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860739+0000","last_change":"2024-05-09T05:00:12.840693+0000","last_active":"2024-05-09T05:00:17.860739+0000","last_peered":"2024-05-09T05:00:17.860739+0000","last_clean":"2024-05-09T05:00:17.860739+0000","last_became_active":"2024-05-09T05:00:12.840400+0000","last_became_peered":"2024-05-09T05:00:12.840400+0000","last_unstale":"2024-05-09T05:00:17.860739+0000","last_undegraded":"2024-05-09T05:00:17.860739+0000","last_fullsized":"2024-05-09T05:00:17.860739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860995+0000","last_change":"2024-05-09T05:00:12.841318+0000","last_active":"2024-05-09T05:00:17.860995+0000","last_peered":"2024-05-09T05:00:17.860995+0000","last_clean":"2024-05-09T05:00:17.860995+0000","last_became_active":"2024-05-09T05:00:12.841026+0000","last_became_peered":"2024-05-09T05:00:12.841026+0000","last_unstale":"2024-05-09T05:00:17.860995+0000","last_undegraded":"2024-05-09T05:00:17.860995+0000","last_fullsized":"2024-05-09T05:00:17.860995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271552+0000","last_change":"2024-05-09T05:00:12.852840+0000","last_active":"2024-05-09T05:00:20.271552+0000","last_peered":"2024-05-09T05:00:20.271552+0000","last_clean":"2024-05-09T05:00:20.271552+0000","last_became_active":"2024-05-09T05:00:12.852259+0000","last_became_peered":"2024-05-09T05:00:12.852259+0000","last_unstale":"2024-05-09T05:00:20.271552+0000","last_undegraded":"2024-05-09T05:00:20.271552+0000","last_fullsized":"2024-05-09T05:00:20.271552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:16:58.143294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652120+0000","last_change":"2024-05-09T05:00:12.846320+0000","last_active":"2024-05-09T05:00:18.652120+0000","last_peered":"2024-05-09T05:00:18.652120+0000","last_clean":"2024-05-09T05:00:18.652120+0000","last_became_active":"2024-05-09T05:00:12.845732+0000","last_became_peered":"2024-05-09T05:00:12.845732+0000","last_unstale":"2024-05-09T05:00:18.652120+0000","last_undegraded":"2024-05-09T05:00:18.652120+0000","last_fullsized":"2024-05-09T05:00:18.652120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:33:12.899897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270936+0000","last_change":"2024-05-09T05:00:12.858255+0000","last_active":"2024-05-09T05:00:20.270936+0000","last_peered":"2024-05-09T05:00:20.270936+0000","last_clean":"2024-05-09T05:00:20.270936+0000","last_became_active":"2024-05-09T05:00:12.857971+0000","last_became_peered":"2024-05-09T05:00:12.857971+0000","last_unstale":"2024-05-09T05:00:20.270936+0000","last_undegraded":"2024-05-09T05:00:20.270936+0000","last_fullsized":"2024-05-09T05:00:20.270936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:23.066586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970422+0000","last_change":"2024-05-09T05:00:12.846491+0000","last_active":"2024-05-09T05:00:19.970422+0000","last_peered":"2024-05-09T05:00:19.970422+0000","last_clean":"2024-05-09T05:00:19.970422+0000","last_became_active":"2024-05-09T05:00:12.845643+0000","last_became_peered":"2024-05-09T05:00:12.845643+0000","last_unstale":"2024-05-09T05:00:19.970422+0000","last_undegraded":"2024-05-09T05:00:19.970422+0000","last_fullsized":"2024-05-09T05:00:19.970422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:20:37.102430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270963+0000","last_change":"2024-05-09T05:00:12.854820+0000","last_active":"2024-05-09T05:00:20.270963+0000","last_peered":"2024-05-09T05:00:20.270963+0000","last_clean":"2024-05-09T05:00:20.270963+0000","last_became_active":"2024-05-09T05:00:12.853663+0000","last_became_peered":"2024-05-09T05:00:12.853663+0000","last_unstale":"2024-05-09T05:00:20.270963+0000","last_undegraded":"2024-05-09T05:00:20.270963+0000","last_fullsized":"2024-05-09T05:00:20.270963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:04.977127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970662+0000","last_change":"2024-05-09T05:00:12.855723+0000","last_active":"2024-05-09T05:00:19.970662+0000","last_peered":"2024-05-09T05:00:19.970662+0000","last_clean":"2024-05-09T05:00:19.970662+0000","last_became_active":"2024-05-09T05:00:12.855479+0000","last_became_peered":"2024-05-09T05:00:12.855479+0000","last_unstale":"2024-05-09T05:00:19.970662+0000","last_undegraded":"2024-05-09T05:00:19.970662+0000","last_fullsized":"2024-05-09T05:00:19.970662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:01:08.488591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623805+0000","last_change":"2024-05-09T05:00:12.838168+0000","last_active":"2024-05-09T05:00:19.623805+0000","last_peered":"2024-05-09T05:00:19.623805+0000","last_clean":"2024-05-09T05:00:19.623805+0000","last_became_active":"2024-05-09T05:00:12.837892+0000","last_became_peered":"2024-05-09T05:00:12.837892+0000","last_unstale":"2024-05-09T05:00:19.623805+0000","last_undegraded":"2024-05-09T05:00:19.623805+0000","last_fullsized":"2024-05-09T05:00:19.623805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:30.691332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271359+0000","last_change":"2024-05-09T05:00:12.851393+0000","last_active":"2024-05-09T05:00:20.271359+0000","last_peered":"2024-05-09T05:00:20.271359+0000","last_clean":"2024-05-09T05:00:20.271359+0000","last_became_active":"2024-05-09T05:00:12.851057+0000","last_became_peered":"2024-05-09T05:00:12.851057+0000","last_unstale":"2024-05-09T05:00:20.271359+0000","last_undegraded":"2024-05-09T05:00:20.271359+0000","last_fullsized":"2024-05-09T05:00:20.271359+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271442+0000","last_change":"2024-05-09T05:00:12.853084+0000","last_active":"2024-05-09T05:00:20.271442+0000","last_peered":"2024-05-09T05:00:20.271442+0000","last_clean":"2024-05-09T05:00:20.271442+0000","last_became_active":"2024-05-09T05:00:12.852602+0000","last_became_peered":"2024-05-09T05:00:12.852602+0000","last_unstale":"2024-05-09T05:00:20.271442+0000","last_undegraded":"2024-05-09T05:00:20.271442+0000","last_fullsized":"2024-05-09T05:00:20.271442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:58.178632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:00:35.921 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:00:36.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.059+0000 7fadc5511640 1 Processor -- start 2024-05-09T05:00:36.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.059+0000 7fadc5511640 1 -- start start 2024-05-09T05:00:36.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.060+0000 7fadc5511640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fadc015f300 0x7fadc0072220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:36.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.060+0000 7fadc5511640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fadc006fe20 0x7fadc00702a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:36.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.060+0000 7fadc5511640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fadc00736d0 0x7fadc0073b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:36.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.060+0000 7fadc5511640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fadc0073240 con 0x7fadc006fe20 2024-05-09T05:00:36.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.060+0000 7fadc5511640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fadc0072760 con 0x7fadc00736d0 2024-05-09T05:00:36.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.060+0000 7fadc5511640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fadc00d08e0 con 0x7fadc015f300 2024-05-09T05:00:36.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.060+0000 7fadbe7fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fadc006fe20 0x7fadc00702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:36.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.060+0000 7fadbe7fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fadc006fe20 0x7fadc00702a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35114/0 (socket says 172.21.15.60:35114) 2024-05-09T05:00:36.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.060+0000 7fadbe7fc640 1 -- 172.21.15.60:0/742061410 learned_addr learned my addr 172.21.15.60:0/742061410 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:36.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.060+0000 7fadbf7fe640 1 --2- 172.21.15.60:0/742061410 >> v2:172.21.15.114:3300/0 conn(0x7fadc00736d0 0x7fadc0073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:36.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.060+0000 7fadbeffd640 1 --2- 172.21.15.60:0/742061410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fadc015f300 0x7fadc0072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:36.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.060+0000 7fadbe7fc640 1 -- 172.21.15.60:0/742061410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fadc015f300 msgr2=0x7fadc0072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:36.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.060+0000 7fadbe7fc640 1 --2- 172.21.15.60:0/742061410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fadc015f300 0x7fadc0072220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.060+0000 7fadbe7fc640 1 -- 172.21.15.60:0/742061410 >> v2:172.21.15.114:3300/0 conn(0x7fadc00736d0 msgr2=0x7fadc0073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:36.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.060+0000 7fadbe7fc640 1 --2- 172.21.15.60:0/742061410 >> v2:172.21.15.114:3300/0 conn(0x7fadc00736d0 0x7fadc0073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.060+0000 7fadbe7fc640 1 -- 172.21.15.60:0/742061410 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc00d0bc0 con 0x7fadc006fe20 2024-05-09T05:00:36.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.060+0000 7fadbf7fe640 1 --2- 172.21.15.60:0/742061410 >> v2:172.21.15.114:3300/0 conn(0x7fadc00736d0 0x7fadc0073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:36.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.061+0000 7fadbe7fc640 1 --2- 172.21.15.60:0/742061410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadc006fe20 0x7fadc00702a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fadc0071160 tx=0x7fada802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=837d01ebfa20a4a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:36.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.061+0000 7fadbdffb640 1 -- 172.21.15.60:0/742061410 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada8037070 con 0x7fadc006fe20 2024-05-09T05:00:36.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.061+0000 7fadbdffb640 1 -- 172.21.15.60:0/742061410 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fada8032440 con 0x7fadc006fe20 2024-05-09T05:00:36.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.061+0000 7fadbdffb640 1 -- 172.21.15.60:0/742061410 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada8032810 con 0x7fadc006fe20 2024-05-09T05:00:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.062+0000 7fadc5511640 1 -- 172.21.15.60:0/742061410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadc006fe20 msgr2=0x7fadc00702a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.062+0000 7fadc5511640 1 --2- 172.21.15.60:0/742061410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadc006fe20 0x7fadc00702a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fadc0071160 tx=0x7fada802a1e0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.062+0000 7fadc5511640 1 -- 172.21.15.60:0/742061410 shutdown_connections 2024-05-09T05:00:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.062+0000 7fadc5511640 1 --2- 172.21.15.60:0/742061410 >> v2:172.21.15.114:3300/0 conn(0x7fadc00736d0 0x7fadc0073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.062+0000 7fadc5511640 1 --2- 172.21.15.60:0/742061410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadc006fe20 0x7fadc00702a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.062+0000 7fadc5511640 1 --2- 172.21.15.60:0/742061410 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fadc015f300 0x7fadc0072220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.062+0000 7fadc5511640 1 -- 172.21.15.60:0/742061410 >> 172.21.15.60:0/742061410 conn(0x7fadc00b4ef0 msgr2=0x7fadc006e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.062+0000 7fadc5511640 1 -- 172.21.15.60:0/742061410 shutdown_connections 2024-05-09T05:00:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.062+0000 7fadc5511640 1 -- 172.21.15.60:0/742061410 wait complete. 2024-05-09T05:00:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.063+0000 7fadc5511640 1 Processor -- start 2024-05-09T05:00:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.063+0000 7fadc5511640 1 -- start start 2024-05-09T05:00:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.064+0000 7fadc5511640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fadc006fe20 0x7fadc00ac9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.064+0000 7fadc5511640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadc00736d0 0x7fadc01f4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.064+0000 7fadc5511640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadc015f300 0x7fadc009fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.064+0000 7fadc5511640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fadc009c1e0 con 0x7fadc015f300 2024-05-09T05:00:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.064+0000 7fadc5511640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fadc0074070 con 0x7fadc00736d0 2024-05-09T05:00:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.064+0000 7fadc5511640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fadc00d04f0 con 0x7fadc006fe20 2024-05-09T05:00:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.064+0000 7fadbf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadc015f300 0x7fadc009fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:36.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.064+0000 7fadbf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadc015f300 0x7fadc009fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35126/0 (socket says 172.21.15.60:35126) 2024-05-09T05:00:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.064+0000 7fadbf7fe640 1 -- 172.21.15.60:0/553876240 learned_addr learned my addr 172.21.15.60:0/553876240 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.064+0000 7fadbe7fc640 1 --2- 172.21.15.60:0/553876240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadc00736d0 0x7fadc01f4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.064+0000 7fadbeffd640 1 --2- 172.21.15.60:0/553876240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fadc006fe20 0x7fadc00ac9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.065+0000 7fadbf7fe640 1 -- 172.21.15.60:0/553876240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fadc006fe20 msgr2=0x7fadc00ac9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.065+0000 7fadbf7fe640 1 --2- 172.21.15.60:0/553876240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fadc006fe20 0x7fadc00ac9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.065+0000 7fadbf7fe640 1 -- 172.21.15.60:0/553876240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadc00736d0 msgr2=0x7fadc01f4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.065+0000 7fadbf7fe640 1 --2- 172.21.15.60:0/553876240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadc00736d0 0x7fadc01f4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.065+0000 7fadbf7fe640 1 -- 172.21.15.60:0/553876240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc00b3d20 con 0x7fadc015f300 2024-05-09T05:00:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.065+0000 7fadbeffd640 1 --2- 172.21.15.60:0/553876240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fadc006fe20 0x7fadc00ac9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.065+0000 7fadbf7fe640 1 --2- 172.21.15.60:0/553876240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadc015f300 0x7fadc009fa90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fadb8007d60 tx=0x7fadb800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.065+0000 7fadb77fe640 1 -- 172.21.15.60:0/553876240 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb801a070 con 0x7fadc015f300 2024-05-09T05:00:36.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.065+0000 7fadb77fe640 1 -- 172.21.15.60:0/553876240 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadb8003990 con 0x7fadc015f300 2024-05-09T05:00:36.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.065+0000 7fadc5511640 1 -- 172.21.15.60:0/553876240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadc015abe0 con 0x7fadc015f300 2024-05-09T05:00:36.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.065+0000 7fadb77fe640 1 -- 172.21.15.60:0/553876240 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb8003d80 con 0x7fadc015f300 2024-05-09T05:00:36.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.065+0000 7fadc5511640 1 -- 172.21.15.60:0/553876240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadc0160ab0 con 0x7fadc015f300 2024-05-09T05:00:36.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.066+0000 7fadc5511640 1 -- 172.21.15.60:0/553876240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadc0051040 con 0x7fadc015f300 2024-05-09T05:00:36.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.067+0000 7fadb77fe640 1 -- 172.21.15.60:0/553876240 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fadb801d070 con 0x7fadc015f300 2024-05-09T05:00:36.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.067+0000 7fadb77fe640 1 --2- 172.21.15.60:0/553876240 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fada00762d0 0x7fada0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:36.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.067+0000 7fadb77fe640 1 -- 172.21.15.60:0/553876240 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fadb8097af0 con 0x7fadc015f300 2024-05-09T05:00:36.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.067+0000 7fadbeffd640 1 --2- 172.21.15.60:0/553876240 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fada00762d0 0x7fada0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:36.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.068+0000 7fadbeffd640 1 --2- 172.21.15.60:0/553876240 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fada00762d0 0x7fada0078790 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fadb0006fd0 tx=0x7fadb0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:36.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.071+0000 7fadb77fe640 1 -- 172.21.15.60:0/553876240 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fadb8010070 con 0x7fadc015f300 2024-05-09T05:00:36.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.258+0000 7fadc5511640 1 -- 172.21.15.60:0/553876240 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fadc015d620 con 0x7fada00762d0 2024-05-09T05:00:36.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.267+0000 7fadb77fe640 1 -- 172.21.15.60:0/553876240 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332523 (secure 0 0 0) 0x7fadc015d620 con 0x7fada00762d0 2024-05-09T05:00:36.271 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:36.273 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-05-09T05:00:36.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.270+0000 7fadc5511640 1 -- 172.21.15.60:0/553876240 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fada00762d0 msgr2=0x7fada0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:36.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.270+0000 7fadc5511640 1 --2- 172.21.15.60:0/553876240 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fada00762d0 0x7fada0078790 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fadb0006fd0 tx=0x7fadb0008040 comp rx=0 tx=0).stop 2024-05-09T05:00:36.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.270+0000 7fadc5511640 1 -- 172.21.15.60:0/553876240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadc015f300 msgr2=0x7fadc009fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:36.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.270+0000 7fadc5511640 1 --2- 172.21.15.60:0/553876240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadc015f300 0x7fadc009fa90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fadb8007d60 tx=0x7fadb800d4a0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.270+0000 7fadc5511640 1 -- 172.21.15.60:0/553876240 shutdown_connections 2024-05-09T05:00:36.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.270+0000 7fadc5511640 1 --2- 172.21.15.60:0/553876240 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fada00762d0 0x7fada0078790 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.270+0000 7fadc5511640 1 --2- 172.21.15.60:0/553876240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadc015f300 0x7fadc009fa90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.270+0000 7fadc5511640 1 --2- 172.21.15.60:0/553876240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadc00736d0 0x7fadc01f4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.270+0000 7fadc5511640 1 --2- 172.21.15.60:0/553876240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fadc006fe20 0x7fadc00ac9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.270+0000 7fadc5511640 1 -- 172.21.15.60:0/553876240 >> 172.21.15.60:0/553876240 conn(0x7fadc00b4ef0 msgr2=0x7fadc006e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:36.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.270+0000 7fadc5511640 1 -- 172.21.15.60:0/553876240 shutdown_connections 2024-05-09T05:00:36.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.270+0000 7fadc5511640 1 -- 172.21.15.60:0/553876240 wait complete. 2024-05-09T05:00:36.282 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-05-09T05:00:34.916031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223524,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751196,"statfs":{"total":773094113280,"available":772865224704,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003753"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624117+0000","last_change":"2024-05-09T05:00:11.849304+0000","last_active":"2024-05-09T05:00:19.624117+0000","last_peered":"2024-05-09T05:00:19.624117+0000","last_clean":"2024-05-09T05:00:19.624117+0000","last_became_active":"2024-05-09T05:00:11.848977+0000","last_became_peered":"2024-05-09T05:00:11.848977+0000","last_unstale":"2024-05-09T05:00:19.624117+0000","last_undegraded":"2024-05-09T05:00:19.624117+0000","last_fullsized":"2024-05-09T05:00:19.624117+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:48:26.640184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652587+0000","last_change":"2024-05-09T05:00:12.853461+0000","last_active":"2024-05-09T05:00:18.652587+0000","last_peered":"2024-05-09T05:00:18.652587+0000","last_clean":"2024-05-09T05:00:18.652587+0000","last_became_active":"2024-05-09T05:00:12.853171+0000","last_became_peered":"2024-05-09T05:00:12.853171+0000","last_unstale":"2024-05-09T05:00:18.652587+0000","last_undegraded":"2024-05-09T05:00:18.652587+0000","last_fullsized":"2024-05-09T05:00:18.652587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:01:08.488591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652158+0000","last_change":"2024-05-09T05:00:16.883712+0000","last_active":"2024-05-09T05:00:18.652158+0000","last_peered":"2024-05-09T05:00:18.652158+0000","last_clean":"2024-05-09T05:00:18.652158+0000","last_became_active":"2024-05-09T05:00:16.883004+0000","last_became_peered":"2024-05-09T05:00:16.883004+0000","last_unstale":"2024-05-09T05:00:18.652158+0000","last_undegraded":"2024-05-09T05:00:18.652158+0000","last_fullsized":"2024-05-09T05:00:18.652158+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:27.371200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652845+0000","last_change":"2024-05-09T05:00:11.839791+0000","last_active":"2024-05-09T05:00:18.652845+0000","last_peered":"2024-05-09T05:00:18.652845+0000","last_clean":"2024-05-09T05:00:18.652845+0000","last_became_active":"2024-05-09T05:00:11.839531+0000","last_became_peered":"2024-05-09T05:00:11.839531+0000","last_unstale":"2024-05-09T05:00:18.652845+0000","last_undegraded":"2024-05-09T05:00:18.652845+0000","last_fullsized":"2024-05-09T05:00:18.652845+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:23.679256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623978+0000","last_change":"2024-05-09T05:00:12.844866+0000","last_active":"2024-05-09T05:00:19.623978+0000","last_peered":"2024-05-09T05:00:19.623978+0000","last_clean":"2024-05-09T05:00:19.623978+0000","last_became_active":"2024-05-09T05:00:12.844015+0000","last_became_peered":"2024-05-09T05:00:12.844015+0000","last_unstale":"2024-05-09T05:00:19.623978+0000","last_undegraded":"2024-05-09T05:00:19.623978+0000","last_fullsized":"2024-05-09T05:00:19.623978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881734+0000","last_change":"2024-05-09T05:00:16.878021+0000","last_active":"2024-05-09T05:00:19.881734+0000","last_peered":"2024-05-09T05:00:19.881734+0000","last_clean":"2024-05-09T05:00:19.881734+0000","last_became_active":"2024-05-09T05:00:16.877322+0000","last_became_peered":"2024-05-09T05:00:16.877322+0000","last_unstale":"2024-05-09T05:00:19.881734+0000","last_undegraded":"2024-05-09T05:00:19.881734+0000","last_fullsized":"2024-05-09T05:00:19.881734+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:34.719408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270583+0000","last_change":"2024-05-09T05:00:11.852994+0000","last_active":"2024-05-09T05:00:20.270583+0000","last_peered":"2024-05-09T05:00:20.270583+0000","last_clean":"2024-05-09T05:00:20.270583+0000","last_became_active":"2024-05-09T05:00:11.852737+0000","last_became_peered":"2024-05-09T05:00:11.852737+0000","last_unstale":"2024-05-09T05:00:20.270583+0000","last_undegraded":"2024-05-09T05:00:20.270583+0000","last_fullsized":"2024-05-09T05:00:20.270583+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:39.794811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970762+0000","last_change":"2024-05-09T05:00:12.845198+0000","last_active":"2024-05-09T05:00:19.970762+0000","last_peered":"2024-05-09T05:00:19.970762+0000","last_clean":"2024-05-09T05:00:19.970762+0000","last_became_active":"2024-05-09T05:00:12.844003+0000","last_became_peered":"2024-05-09T05:00:12.844003+0000","last_unstale":"2024-05-09T05:00:19.970762+0000","last_undegraded":"2024-05-09T05:00:19.970762+0000","last_fullsized":"2024-05-09T05:00:19.970762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:45.314128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882179+0000","last_change":"2024-05-09T05:00:16.877475+0000","last_active":"2024-05-09T05:00:19.882179+0000","last_peered":"2024-05-09T05:00:19.882179+0000","last_clean":"2024-05-09T05:00:19.882179+0000","last_became_active":"2024-05-09T05:00:16.876947+0000","last_became_peered":"2024-05-09T05:00:16.876947+0000","last_unstale":"2024-05-09T05:00:19.882179+0000","last_undegraded":"2024-05-09T05:00:19.882179+0000","last_fullsized":"2024-05-09T05:00:19.882179+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:49.342204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271162+0000","last_change":"2024-05-09T05:00:11.853073+0000","last_active":"2024-05-09T05:00:20.271162+0000","last_peered":"2024-05-09T05:00:20.271162+0000","last_clean":"2024-05-09T05:00:20.271162+0000","last_became_active":"2024-05-09T05:00:11.852759+0000","last_became_peered":"2024-05-09T05:00:11.852759+0000","last_unstale":"2024-05-09T05:00:20.271162+0000","last_undegraded":"2024-05-09T05:00:20.271162+0000","last_fullsized":"2024-05-09T05:00:20.271162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:01.738771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970943+0000","last_change":"2024-05-09T05:00:12.844419+0000","last_active":"2024-05-09T05:00:19.970943+0000","last_peered":"2024-05-09T05:00:19.970943+0000","last_clean":"2024-05-09T05:00:19.970943+0000","last_became_active":"2024-05-09T05:00:12.843951+0000","last_became_peered":"2024-05-09T05:00:12.843951+0000","last_unstale":"2024-05-09T05:00:19.970943+0000","last_undegraded":"2024-05-09T05:00:19.970943+0000","last_fullsized":"2024-05-09T05:00:19.970943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:23.066586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881928+0000","last_change":"2024-05-09T05:00:16.875575+0000","last_active":"2024-05-09T05:00:19.881928+0000","last_peered":"2024-05-09T05:00:19.881928+0000","last_clean":"2024-05-09T05:00:19.881928+0000","last_became_active":"2024-05-09T05:00:16.874993+0000","last_became_peered":"2024-05-09T05:00:16.874993+0000","last_unstale":"2024-05-09T05:00:19.881928+0000","last_undegraded":"2024-05-09T05:00:19.881928+0000","last_fullsized":"2024-05-09T05:00:19.881928+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:34.414100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860116+0000","last_change":"2024-05-09T05:00:12.844310+0000","last_active":"2024-05-09T05:00:17.860116+0000","last_peered":"2024-05-09T05:00:17.860116+0000","last_clean":"2024-05-09T05:00:17.860116+0000","last_became_active":"2024-05-09T05:00:12.843870+0000","last_became_peered":"2024-05-09T05:00:12.843870+0000","last_unstale":"2024-05-09T05:00:17.860116+0000","last_undegraded":"2024-05-09T05:00:17.860116+0000","last_fullsized":"2024-05-09T05:00:17.860116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:19.080163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271207+0000","last_change":"2024-05-09T05:00:11.849654+0000","last_active":"2024-05-09T05:00:20.271207+0000","last_peered":"2024-05-09T05:00:20.271207+0000","last_clean":"2024-05-09T05:00:20.271207+0000","last_became_active":"2024-05-09T05:00:11.849112+0000","last_became_peered":"2024-05-09T05:00:11.849112+0000","last_unstale":"2024-05-09T05:00:20.271207+0000","last_undegraded":"2024-05-09T05:00:20.271207+0000","last_fullsized":"2024-05-09T05:00:20.271207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:04:53.801780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882282+0000","last_change":"2024-05-09T05:00:16.874883+0000","last_active":"2024-05-09T05:00:19.882282+0000","last_peered":"2024-05-09T05:00:19.882282+0000","last_clean":"2024-05-09T05:00:19.882282+0000","last_became_active":"2024-05-09T05:00:16.874390+0000","last_became_peered":"2024-05-09T05:00:16.874390+0000","last_unstale":"2024-05-09T05:00:19.882282+0000","last_undegraded":"2024-05-09T05:00:19.882282+0000","last_fullsized":"2024-05-09T05:00:19.882282+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:09.005203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652613+0000","last_change":"2024-05-09T05:00:12.855440+0000","last_active":"2024-05-09T05:00:18.652613+0000","last_peered":"2024-05-09T05:00:18.652613+0000","last_clean":"2024-05-09T05:00:18.652613+0000","last_became_active":"2024-05-09T05:00:12.854274+0000","last_became_peered":"2024-05-09T05:00:12.854274+0000","last_unstale":"2024-05-09T05:00:18.652613+0000","last_undegraded":"2024-05-09T05:00:18.652613+0000","last_fullsized":"2024-05-09T05:00:18.652613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:30.691332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270354+0000","last_change":"2024-05-09T05:00:11.842122+0000","last_active":"2024-05-09T05:00:20.270354+0000","last_peered":"2024-05-09T05:00:20.270354+0000","last_clean":"2024-05-09T05:00:20.270354+0000","last_became_active":"2024-05-09T05:00:11.840835+0000","last_became_peered":"2024-05-09T05:00:11.840835+0000","last_unstale":"2024-05-09T05:00:20.270354+0000","last_undegraded":"2024-05-09T05:00:20.270354+0000","last_fullsized":"2024-05-09T05:00:20.270354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:41:27.364045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882288+0000","last_change":"2024-05-09T05:00:16.875354+0000","last_active":"2024-05-09T05:00:19.882288+0000","last_peered":"2024-05-09T05:00:19.882288+0000","last_clean":"2024-05-09T05:00:19.882288+0000","last_became_active":"2024-05-09T05:00:16.874942+0000","last_became_peered":"2024-05-09T05:00:16.874942+0000","last_unstale":"2024-05-09T05:00:19.882288+0000","last_undegraded":"2024-05-09T05:00:19.882288+0000","last_fullsized":"2024-05-09T05:00:19.882288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:27.094662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652325+0000","last_change":"2024-05-09T05:00:12.853552+0000","last_active":"2024-05-09T05:00:18.652325+0000","last_peered":"2024-05-09T05:00:18.652325+0000","last_clean":"2024-05-09T05:00:18.652325+0000","last_became_active":"2024-05-09T05:00:12.853268+0000","last_became_peered":"2024-05-09T05:00:12.853268+0000","last_unstale":"2024-05-09T05:00:18.652325+0000","last_undegraded":"2024-05-09T05:00:18.652325+0000","last_fullsized":"2024-05-09T05:00:18.652325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:44.359533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270643+0000","last_change":"2024-05-09T05:00:11.848638+0000","last_active":"2024-05-09T05:00:20.270643+0000","last_peered":"2024-05-09T05:00:20.270643+0000","last_clean":"2024-05-09T05:00:20.270643+0000","last_became_active":"2024-05-09T05:00:11.848351+0000","last_became_peered":"2024-05-09T05:00:11.848351+0000","last_unstale":"2024-05-09T05:00:20.270643+0000","last_undegraded":"2024-05-09T05:00:20.270643+0000","last_fullsized":"2024-05-09T05:00:20.270643+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:01.738771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271321+0000","last_change":"2024-05-09T05:00:16.880744+0000","last_active":"2024-05-09T05:00:20.271321+0000","last_peered":"2024-05-09T05:00:20.271321+0000","last_clean":"2024-05-09T05:00:20.271321+0000","last_became_active":"2024-05-09T05:00:16.880373+0000","last_became_peered":"2024-05-09T05:00:16.880373+0000","last_unstale":"2024-05-09T05:00:20.271321+0000","last_undegraded":"2024-05-09T05:00:20.271321+0000","last_fullsized":"2024-05-09T05:00:20.271321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:10.821888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882019+0000","last_change":"2024-05-09T05:00:12.835034+0000","last_active":"2024-05-09T05:00:19.882019+0000","last_peered":"2024-05-09T05:00:19.882019+0000","last_clean":"2024-05-09T05:00:19.882019+0000","last_became_active":"2024-05-09T05:00:12.834744+0000","last_became_peered":"2024-05-09T05:00:12.834744+0000","last_unstale":"2024-05-09T05:00:19.882019+0000","last_undegraded":"2024-05-09T05:00:19.882019+0000","last_fullsized":"2024-05-09T05:00:19.882019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:26.805203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270527+0000","last_change":"2024-05-09T05:00:11.848682+0000","last_active":"2024-05-09T05:00:20.270527+0000","last_peered":"2024-05-09T05:00:20.270527+0000","last_clean":"2024-05-09T05:00:20.270527+0000","last_became_active":"2024-05-09T05:00:11.848396+0000","last_became_peered":"2024-05-09T05:00:11.848396+0000","last_unstale":"2024-05-09T05:00:20.270527+0000","last_undegraded":"2024-05-09T05:00:20.270527+0000","last_fullsized":"2024-05-09T05:00:20.270527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:44:46.364938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882265+0000","last_change":"2024-05-09T05:00:16.877861+0000","last_active":"2024-05-09T05:00:19.882265+0000","last_peered":"2024-05-09T05:00:19.882265+0000","last_clean":"2024-05-09T05:00:19.882265+0000","last_became_active":"2024-05-09T05:00:16.877155+0000","last_became_peered":"2024-05-09T05:00:16.877155+0000","last_unstale":"2024-05-09T05:00:19.882265+0000","last_undegraded":"2024-05-09T05:00:19.882265+0000","last_fullsized":"2024-05-09T05:00:19.882265+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:01:12.516667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624129+0000","last_change":"2024-05-09T05:00:12.852207+0000","last_active":"2024-05-09T05:00:19.624129+0000","last_peered":"2024-05-09T05:00:19.624129+0000","last_clean":"2024-05-09T05:00:19.624129+0000","last_became_active":"2024-05-09T05:00:12.851927+0000","last_became_peered":"2024-05-09T05:00:12.851927+0000","last_unstale":"2024-05-09T05:00:19.624129+0000","last_undegraded":"2024-05-09T05:00:19.624129+0000","last_fullsized":"2024-05-09T05:00:19.624129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:44.359533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.861099+0000","last_change":"2024-05-09T05:00:11.849008+0000","last_active":"2024-05-09T05:00:17.861099+0000","last_peered":"2024-05-09T05:00:17.861099+0000","last_clean":"2024-05-09T05:00:17.861099+0000","last_became_active":"2024-05-09T05:00:11.848741+0000","last_became_peered":"2024-05-09T05:00:11.848741+0000","last_unstale":"2024-05-09T05:00:17.861099+0000","last_undegraded":"2024-05-09T05:00:17.861099+0000","last_fullsized":"2024-05-09T05:00:17.861099+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:39.794811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881982+0000","last_change":"2024-05-09T05:00:16.879683+0000","last_active":"2024-05-09T05:00:19.881982+0000","last_peered":"2024-05-09T05:00:19.881982+0000","last_clean":"2024-05-09T05:00:19.881982+0000","last_became_active":"2024-05-09T05:00:16.879337+0000","last_became_peered":"2024-05-09T05:00:16.879337+0000","last_unstale":"2024-05-09T05:00:19.881982+0000","last_undegraded":"2024-05-09T05:00:19.881982+0000","last_fullsized":"2024-05-09T05:00:19.881982+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:48.387609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652854+0000","last_change":"2024-05-09T05:00:12.852575+0000","last_active":"2024-05-09T05:00:18.652854+0000","last_peered":"2024-05-09T05:00:18.652854+0000","last_clean":"2024-05-09T05:00:18.652854+0000","last_became_active":"2024-05-09T05:00:12.852308+0000","last_became_peered":"2024-05-09T05:00:12.852308+0000","last_unstale":"2024-05-09T05:00:18.652854+0000","last_undegraded":"2024-05-09T05:00:18.652854+0000","last_fullsized":"2024-05-09T05:00:18.652854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270373+0000","last_change":"2024-05-09T05:00:11.839637+0000","last_active":"2024-05-09T05:00:20.270373+0000","last_peered":"2024-05-09T05:00:20.270373+0000","last_clean":"2024-05-09T05:00:20.270373+0000","last_became_active":"2024-05-09T05:00:11.839214+0000","last_became_peered":"2024-05-09T05:00:11.839214+0000","last_unstale":"2024-05-09T05:00:20.270373+0000","last_undegraded":"2024-05-09T05:00:20.270373+0000","last_fullsized":"2024-05-09T05:00:20.270373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:25.803010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652033+0000","last_change":"2024-05-09T05:00:16.883218+0000","last_active":"2024-05-09T05:00:18.652033+0000","last_peered":"2024-05-09T05:00:18.652033+0000","last_clean":"2024-05-09T05:00:18.652033+0000","last_became_active":"2024-05-09T05:00:16.882353+0000","last_became_peered":"2024-05-09T05:00:16.882353+0000","last_unstale":"2024-05-09T05:00:18.652033+0000","last_undegraded":"2024-05-09T05:00:18.652033+0000","last_fullsized":"2024-05-09T05:00:18.652033+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:02.171370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271136+0000","last_change":"2024-05-09T05:00:12.846656+0000","last_active":"2024-05-09T05:00:20.271136+0000","last_peered":"2024-05-09T05:00:20.271136+0000","last_clean":"2024-05-09T05:00:20.271136+0000","last_became_active":"2024-05-09T05:00:12.846359+0000","last_became_peered":"2024-05-09T05:00:12.846359+0000","last_unstale":"2024-05-09T05:00:20.271136+0000","last_undegraded":"2024-05-09T05:00:20.271136+0000","last_fullsized":"2024-05-09T05:00:20.271136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:49.330395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270574+0000","last_change":"2024-05-09T05:00:11.850170+0000","last_active":"2024-05-09T05:00:20.270574+0000","last_peered":"2024-05-09T05:00:20.270574+0000","last_clean":"2024-05-09T05:00:20.270574+0000","last_became_active":"2024-05-09T05:00:11.849866+0000","last_became_peered":"2024-05-09T05:00:11.849866+0000","last_unstale":"2024-05-09T05:00:20.270574+0000","last_undegraded":"2024-05-09T05:00:20.270574+0000","last_fullsized":"2024-05-09T05:00:20.270574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:27:57.420062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623671+0000","last_change":"2024-05-09T05:00:16.879852+0000","last_active":"2024-05-09T05:00:19.623671+0000","last_peered":"2024-05-09T05:00:19.623671+0000","last_clean":"2024-05-09T05:00:19.623671+0000","last_became_active":"2024-05-09T05:00:16.878219+0000","last_became_peered":"2024-05-09T05:00:16.878219+0000","last_unstale":"2024-05-09T05:00:19.623671+0000","last_undegraded":"2024-05-09T05:00:19.623671+0000","last_fullsized":"2024-05-09T05:00:19.623671+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:20.415639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.971002+0000","last_change":"2024-05-09T05:00:12.859874+0000","last_active":"2024-05-09T05:00:19.971002+0000","last_peered":"2024-05-09T05:00:19.971002+0000","last_clean":"2024-05-09T05:00:19.971002+0000","last_became_active":"2024-05-09T05:00:12.859620+0000","last_became_peered":"2024-05-09T05:00:12.859620+0000","last_unstale":"2024-05-09T05:00:19.971002+0000","last_undegraded":"2024-05-09T05:00:19.971002+0000","last_fullsized":"2024-05-09T05:00:19.971002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:04.977127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270358+0000","last_change":"2024-05-09T05:00:11.838593+0000","last_active":"2024-05-09T05:00:20.270358+0000","last_peered":"2024-05-09T05:00:20.270358+0000","last_clean":"2024-05-09T05:00:20.270358+0000","last_became_active":"2024-05-09T05:00:11.838314+0000","last_became_peered":"2024-05-09T05:00:11.838314+0000","last_unstale":"2024-05-09T05:00:20.270358+0000","last_undegraded":"2024-05-09T05:00:20.270358+0000","last_fullsized":"2024-05-09T05:00:20.270358+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:49.660089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270704+0000","last_change":"2024-05-09T05:00:16.876934+0000","last_active":"2024-05-09T05:00:20.270704+0000","last_peered":"2024-05-09T05:00:20.270704+0000","last_clean":"2024-05-09T05:00:20.270704+0000","last_became_active":"2024-05-09T05:00:16.876531+0000","last_became_peered":"2024-05-09T05:00:16.876531+0000","last_unstale":"2024-05-09T05:00:20.270704+0000","last_undegraded":"2024-05-09T05:00:20.270704+0000","last_fullsized":"2024-05-09T05:00:20.270704+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:54.996456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882435+0000","last_change":"2024-05-09T05:00:12.849957+0000","last_active":"2024-05-09T05:00:19.882435+0000","last_peered":"2024-05-09T05:00:19.882435+0000","last_clean":"2024-05-09T05:00:19.882435+0000","last_became_active":"2024-05-09T05:00:12.849138+0000","last_became_peered":"2024-05-09T05:00:12.849138+0000","last_unstale":"2024-05-09T05:00:19.882435+0000","last_undegraded":"2024-05-09T05:00:19.882435+0000","last_fullsized":"2024-05-09T05:00:19.882435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:02.740965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970665+0000","last_change":"2024-05-09T05:00:11.840734+0000","last_active":"2024-05-09T05:00:19.970665+0000","last_peered":"2024-05-09T05:00:19.970665+0000","last_clean":"2024-05-09T05:00:19.970665+0000","last_became_active":"2024-05-09T05:00:11.840463+0000","last_became_peered":"2024-05-09T05:00:11.840463+0000","last_unstale":"2024-05-09T05:00:19.970665+0000","last_undegraded":"2024-05-09T05:00:19.970665+0000","last_fullsized":"2024-05-09T05:00:19.970665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:49.660089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271060+0000","last_change":"2024-05-09T05:00:16.875120+0000","last_active":"2024-05-09T05:00:20.271060+0000","last_peered":"2024-05-09T05:00:20.271060+0000","last_clean":"2024-05-09T05:00:20.271060+0000","last_became_active":"2024-05-09T05:00:16.874728+0000","last_became_peered":"2024-05-09T05:00:16.874728+0000","last_unstale":"2024-05-09T05:00:20.271060+0000","last_undegraded":"2024-05-09T05:00:20.271060+0000","last_fullsized":"2024-05-09T05:00:20.271060+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:20.415639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271161+0000","last_change":"2024-05-09T05:00:12.851489+0000","last_active":"2024-05-09T05:00:20.271161+0000","last_peered":"2024-05-09T05:00:20.271161+0000","last_clean":"2024-05-09T05:00:20.271161+0000","last_became_active":"2024-05-09T05:00:12.841469+0000","last_became_peered":"2024-05-09T05:00:12.841469+0000","last_unstale":"2024-05-09T05:00:20.271161+0000","last_undegraded":"2024-05-09T05:00:20.271161+0000","last_fullsized":"2024-05-09T05:00:20.271161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:45.463031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860713+0000","last_change":"2024-05-09T05:00:11.841055+0000","last_active":"2024-05-09T05:00:17.860713+0000","last_peered":"2024-05-09T05:00:17.860713+0000","last_clean":"2024-05-09T05:00:17.860713+0000","last_became_active":"2024-05-09T05:00:11.839965+0000","last_became_peered":"2024-05-09T05:00:11.839965+0000","last_unstale":"2024-05-09T05:00:17.860713+0000","last_undegraded":"2024-05-09T05:00:17.860713+0000","last_fullsized":"2024-05-09T05:00:17.860713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:41:27.364045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271386+0000","last_change":"2024-05-09T05:00:16.873524+0000","last_active":"2024-05-09T05:00:20.271386+0000","last_peered":"2024-05-09T05:00:20.271386+0000","last_clean":"2024-05-09T05:00:20.271386+0000","last_became_active":"2024-05-09T05:00:16.872894+0000","last_became_peered":"2024-05-09T05:00:16.872894+0000","last_unstale":"2024-05-09T05:00:20.271386+0000","last_undegraded":"2024-05-09T05:00:20.271386+0000","last_fullsized":"2024-05-09T05:00:20.271386+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:32:00.425543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623506+0000","last_change":"2024-05-09T05:00:12.849280+0000","last_active":"2024-05-09T05:00:19.623506+0000","last_peered":"2024-05-09T05:00:19.623506+0000","last_clean":"2024-05-09T05:00:19.623506+0000","last_became_active":"2024-05-09T05:00:12.848968+0000","last_became_peered":"2024-05-09T05:00:12.848968+0000","last_unstale":"2024-05-09T05:00:19.623506+0000","last_undegraded":"2024-05-09T05:00:19.623506+0000","last_fullsized":"2024-05-09T05:00:19.623506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:49.330395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270849+0000","last_change":"2024-05-09T05:00:11.852903+0000","last_active":"2024-05-09T05:00:20.270849+0000","last_peered":"2024-05-09T05:00:20.270849+0000","last_clean":"2024-05-09T05:00:20.270849+0000","last_became_active":"2024-05-09T05:00:11.852584+0000","last_became_peered":"2024-05-09T05:00:11.852584+0000","last_unstale":"2024-05-09T05:00:20.270849+0000","last_undegraded":"2024-05-09T05:00:20.270849+0000","last_fullsized":"2024-05-09T05:00:20.270849+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:29:13.766278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624019+0000","last_change":"2024-05-09T05:00:16.882378+0000","last_active":"2024-05-09T05:00:19.624019+0000","last_peered":"2024-05-09T05:00:19.624019+0000","last_clean":"2024-05-09T05:00:19.624019+0000","last_became_active":"2024-05-09T05:00:16.881947+0000","last_became_peered":"2024-05-09T05:00:16.881947+0000","last_unstale":"2024-05-09T05:00:19.624019+0000","last_undegraded":"2024-05-09T05:00:19.624019+0000","last_fullsized":"2024-05-09T05:00:19.624019+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:32:00.425543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652370+0000","last_change":"2024-05-09T05:00:12.847632+0000","last_active":"2024-05-09T05:00:18.652370+0000","last_peered":"2024-05-09T05:00:18.652370+0000","last_clean":"2024-05-09T05:00:18.652370+0000","last_became_active":"2024-05-09T05:00:12.847358+0000","last_became_peered":"2024-05-09T05:00:12.847358+0000","last_unstale":"2024-05-09T05:00:18.652370+0000","last_undegraded":"2024-05-09T05:00:18.652370+0000","last_fullsized":"2024-05-09T05:00:18.652370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:49.330395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970011+0000","last_change":"2024-05-09T05:00:11.842747+0000","last_active":"2024-05-09T05:00:19.970011+0000","last_peered":"2024-05-09T05:00:19.970011+0000","last_clean":"2024-05-09T05:00:19.970011+0000","last_became_active":"2024-05-09T05:00:11.842474+0000","last_became_peered":"2024-05-09T05:00:11.842474+0000","last_unstale":"2024-05-09T05:00:19.970011+0000","last_undegraded":"2024-05-09T05:00:19.970011+0000","last_fullsized":"2024-05-09T05:00:19.970011+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:01.738771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":118,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271122+0000","last_change":"2024-05-09T05:00:08.667158+0000","last_active":"2024-05-09T05:00:20.271122+0000","last_peered":"2024-05-09T05:00:20.271122+0000","last_clean":"2024-05-09T05:00:20.271122+0000","last_became_active":"2024-05-09T05:00:08.666316+0000","last_became_peered":"2024-05-09T05:00:08.666316+0000","last_unstale":"2024-05-09T05:00:20.271122+0000","last_undegraded":"2024-05-09T05:00:20.271122+0000","last_fullsized":"2024-05-09T05:00:20.271122+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_clean_scrub_stamp":"2024-05-09T05:00:07.370119+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:20.234643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860739+0000","last_change":"2024-05-09T05:00:16.880367+0000","last_active":"2024-05-09T05:00:17.860739+0000","last_peered":"2024-05-09T05:00:17.860739+0000","last_clean":"2024-05-09T05:00:17.860739+0000","last_became_active":"2024-05-09T05:00:16.879980+0000","last_became_peered":"2024-05-09T05:00:16.879980+0000","last_unstale":"2024-05-09T05:00:17.860739+0000","last_undegraded":"2024-05-09T05:00:17.860739+0000","last_fullsized":"2024-05-09T05:00:17.860739+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:54.996456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271273+0000","last_change":"2024-05-09T05:00:12.852180+0000","last_active":"2024-05-09T05:00:20.271273+0000","last_peered":"2024-05-09T05:00:20.271273+0000","last_clean":"2024-05-09T05:00:20.271273+0000","last_became_active":"2024-05-09T05:00:12.851828+0000","last_became_peered":"2024-05-09T05:00:12.851828+0000","last_unstale":"2024-05-09T05:00:20.271273+0000","last_undegraded":"2024-05-09T05:00:20.271273+0000","last_fullsized":"2024-05-09T05:00:20.271273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:16.387563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.861065+0000","last_change":"2024-05-09T05:00:11.843024+0000","last_active":"2024-05-09T05:00:17.861065+0000","last_peered":"2024-05-09T05:00:17.861065+0000","last_clean":"2024-05-09T05:00:17.861065+0000","last_became_active":"2024-05-09T05:00:11.842703+0000","last_became_peered":"2024-05-09T05:00:11.842703+0000","last_unstale":"2024-05-09T05:00:17.861065+0000","last_undegraded":"2024-05-09T05:00:17.861065+0000","last_fullsized":"2024-05-09T05:00:17.861065+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:48:26.640184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860188+0000","last_change":"2024-05-09T05:00:16.879385+0000","last_active":"2024-05-09T05:00:17.860188+0000","last_peered":"2024-05-09T05:00:17.860188+0000","last_clean":"2024-05-09T05:00:17.860188+0000","last_became_active":"2024-05-09T05:00:16.878956+0000","last_became_peered":"2024-05-09T05:00:16.878956+0000","last_unstale":"2024-05-09T05:00:17.860188+0000","last_undegraded":"2024-05-09T05:00:17.860188+0000","last_fullsized":"2024-05-09T05:00:17.860188+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:49.342204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970802+0000","last_change":"2024-05-09T05:00:12.848110+0000","last_active":"2024-05-09T05:00:19.970802+0000","last_peered":"2024-05-09T05:00:19.970802+0000","last_clean":"2024-05-09T05:00:19.970802+0000","last_became_active":"2024-05-09T05:00:12.847628+0000","last_became_peered":"2024-05-09T05:00:12.847628+0000","last_unstale":"2024-05-09T05:00:19.970802+0000","last_undegraded":"2024-05-09T05:00:19.970802+0000","last_fullsized":"2024-05-09T05:00:19.970802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:16.387563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271019+0000","last_change":"2024-05-09T05:00:11.845441+0000","last_active":"2024-05-09T05:00:20.271019+0000","last_peered":"2024-05-09T05:00:20.271019+0000","last_clean":"2024-05-09T05:00:20.271019+0000","last_became_active":"2024-05-09T05:00:11.845151+0000","last_became_peered":"2024-05-09T05:00:11.845151+0000","last_unstale":"2024-05-09T05:00:20.271019+0000","last_undegraded":"2024-05-09T05:00:20.271019+0000","last_fullsized":"2024-05-09T05:00:20.271019+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:39.794811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623840+0000","last_change":"2024-05-09T05:00:16.880872+0000","last_active":"2024-05-09T05:00:19.623840+0000","last_peered":"2024-05-09T05:00:19.623840+0000","last_clean":"2024-05-09T05:00:19.623840+0000","last_became_active":"2024-05-09T05:00:16.876853+0000","last_became_peered":"2024-05-09T05:00:16.876853+0000","last_unstale":"2024-05-09T05:00:19.623840+0000","last_undegraded":"2024-05-09T05:00:19.623840+0000","last_fullsized":"2024-05-09T05:00:19.623840+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:47:02.206708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271556+0000","last_change":"2024-05-09T05:00:12.854694+0000","last_active":"2024-05-09T05:00:20.271556+0000","last_peered":"2024-05-09T05:00:20.271556+0000","last_clean":"2024-05-09T05:00:20.271556+0000","last_became_active":"2024-05-09T05:00:12.854098+0000","last_became_peered":"2024-05-09T05:00:12.854098+0000","last_unstale":"2024-05-09T05:00:20.271556+0000","last_undegraded":"2024-05-09T05:00:20.271556+0000","last_fullsized":"2024-05-09T05:00:20.271556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:23.343124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860998+0000","last_change":"2024-05-09T05:00:11.835134+0000","last_active":"2024-05-09T05:00:17.860998+0000","last_peered":"2024-05-09T05:00:17.860998+0000","last_clean":"2024-05-09T05:00:17.860998+0000","last_became_active":"2024-05-09T05:00:11.834773+0000","last_became_peered":"2024-05-09T05:00:11.834773+0000","last_unstale":"2024-05-09T05:00:17.860998+0000","last_undegraded":"2024-05-09T05:00:17.860998+0000","last_fullsized":"2024-05-09T05:00:17.860998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:50.824636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970193+0000","last_change":"2024-05-09T05:00:16.879633+0000","last_active":"2024-05-09T05:00:19.970193+0000","last_peered":"2024-05-09T05:00:19.970193+0000","last_clean":"2024-05-09T05:00:19.970193+0000","last_became_active":"2024-05-09T05:00:16.879232+0000","last_became_peered":"2024-05-09T05:00:16.879232+0000","last_unstale":"2024-05-09T05:00:19.970193+0000","last_undegraded":"2024-05-09T05:00:19.970193+0000","last_fullsized":"2024-05-09T05:00:19.970193+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:18:15.413800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271558+0000","last_change":"2024-05-09T05:00:12.852679+0000","last_active":"2024-05-09T05:00:20.271558+0000","last_peered":"2024-05-09T05:00:20.271558+0000","last_clean":"2024-05-09T05:00:20.271558+0000","last_became_active":"2024-05-09T05:00:12.852434+0000","last_became_peered":"2024-05-09T05:00:12.852434+0000","last_unstale":"2024-05-09T05:00:20.271558+0000","last_undegraded":"2024-05-09T05:00:20.271558+0000","last_fullsized":"2024-05-09T05:00:20.271558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:06.793812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881482+0000","last_change":"2024-05-09T05:00:11.845447+0000","last_active":"2024-05-09T05:00:19.881482+0000","last_peered":"2024-05-09T05:00:19.881482+0000","last_clean":"2024-05-09T05:00:19.881482+0000","last_became_active":"2024-05-09T05:00:11.845139+0000","last_became_peered":"2024-05-09T05:00:11.845139+0000","last_unstale":"2024-05-09T05:00:19.881482+0000","last_undegraded":"2024-05-09T05:00:19.881482+0000","last_fullsized":"2024-05-09T05:00:19.881482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:32:22.225189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.881692+0000","last_change":"2024-05-09T05:00:16.877859+0000","last_active":"2024-05-09T05:00:19.881692+0000","last_peered":"2024-05-09T05:00:19.881692+0000","last_clean":"2024-05-09T05:00:19.881692+0000","last_became_active":"2024-05-09T05:00:16.877418+0000","last_became_peered":"2024-05-09T05:00:16.877418+0000","last_unstale":"2024-05-09T05:00:19.881692+0000","last_undegraded":"2024-05-09T05:00:19.881692+0000","last_fullsized":"2024-05-09T05:00:19.881692+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:28.924332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271116+0000","last_change":"2024-05-09T05:00:12.855313+0000","last_active":"2024-05-09T05:00:20.271116+0000","last_peered":"2024-05-09T05:00:20.271116+0000","last_clean":"2024-05-09T05:00:20.271116+0000","last_became_active":"2024-05-09T05:00:12.855041+0000","last_became_peered":"2024-05-09T05:00:12.855041+0000","last_unstale":"2024-05-09T05:00:20.271116+0000","last_undegraded":"2024-05-09T05:00:20.271116+0000","last_fullsized":"2024-05-09T05:00:20.271116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:01:08.488591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860579+0000","last_change":"2024-05-09T05:00:11.839443+0000","last_active":"2024-05-09T05:00:17.860579+0000","last_peered":"2024-05-09T05:00:17.860579+0000","last_clean":"2024-05-09T05:00:17.860579+0000","last_became_active":"2024-05-09T05:00:11.839169+0000","last_became_peered":"2024-05-09T05:00:11.839169+0000","last_unstale":"2024-05-09T05:00:17.860579+0000","last_undegraded":"2024-05-09T05:00:17.860579+0000","last_fullsized":"2024-05-09T05:00:17.860579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:10.779395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623875+0000","last_change":"2024-05-09T05:00:16.876207+0000","last_active":"2024-05-09T05:00:19.623875+0000","last_peered":"2024-05-09T05:00:19.623875+0000","last_clean":"2024-05-09T05:00:19.623875+0000","last_became_active":"2024-05-09T05:00:16.875739+0000","last_became_peered":"2024-05-09T05:00:16.875739+0000","last_unstale":"2024-05-09T05:00:19.623875+0000","last_undegraded":"2024-05-09T05:00:19.623875+0000","last_fullsized":"2024-05-09T05:00:19.623875+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:10.821888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860230+0000","last_change":"2024-05-09T05:00:12.851872+0000","last_active":"2024-05-09T05:00:17.860230+0000","last_peered":"2024-05-09T05:00:17.860230+0000","last_clean":"2024-05-09T05:00:17.860230+0000","last_became_active":"2024-05-09T05:00:12.851590+0000","last_became_peered":"2024-05-09T05:00:12.851590+0000","last_unstale":"2024-05-09T05:00:17.860230+0000","last_undegraded":"2024-05-09T05:00:17.860230+0000","last_fullsized":"2024-05-09T05:00:17.860230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:49.330395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860330+0000","last_change":"2024-05-09T05:00:11.845978+0000","last_active":"2024-05-09T05:00:17.860330+0000","last_peered":"2024-05-09T05:00:17.860330+0000","last_clean":"2024-05-09T05:00:17.860330+0000","last_became_active":"2024-05-09T05:00:11.845129+0000","last_became_peered":"2024-05-09T05:00:11.845129+0000","last_unstale":"2024-05-09T05:00:17.860330+0000","last_undegraded":"2024-05-09T05:00:17.860330+0000","last_fullsized":"2024-05-09T05:00:17.860330+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:49.660089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652710+0000","last_change":"2024-05-09T05:00:16.882756+0000","last_active":"2024-05-09T05:00:18.652710+0000","last_peered":"2024-05-09T05:00:18.652710+0000","last_clean":"2024-05-09T05:00:18.652710+0000","last_became_active":"2024-05-09T05:00:16.882378+0000","last_became_peered":"2024-05-09T05:00:16.882378+0000","last_unstale":"2024-05-09T05:00:18.652710+0000","last_undegraded":"2024-05-09T05:00:18.652710+0000","last_fullsized":"2024-05-09T05:00:18.652710+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:51:44.393878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271158+0000","last_change":"2024-05-09T05:00:12.854634+0000","last_active":"2024-05-09T05:00:20.271158+0000","last_peered":"2024-05-09T05:00:20.271158+0000","last_clean":"2024-05-09T05:00:20.271158+0000","last_became_active":"2024-05-09T05:00:12.854366+0000","last_became_peered":"2024-05-09T05:00:12.854366+0000","last_unstale":"2024-05-09T05:00:20.271158+0000","last_undegraded":"2024-05-09T05:00:20.271158+0000","last_fullsized":"2024-05-09T05:00:20.271158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:45.314128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.969990+0000","last_change":"2024-05-09T05:00:11.843045+0000","last_active":"2024-05-09T05:00:19.969990+0000","last_peered":"2024-05-09T05:00:19.969990+0000","last_clean":"2024-05-09T05:00:19.969990+0000","last_became_active":"2024-05-09T05:00:11.842728+0000","last_became_peered":"2024-05-09T05:00:11.842728+0000","last_unstale":"2024-05-09T05:00:19.969990+0000","last_undegraded":"2024-05-09T05:00:19.969990+0000","last_fullsized":"2024-05-09T05:00:19.969990+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:25.803010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652628+0000","last_change":"2024-05-09T05:00:16.880672+0000","last_active":"2024-05-09T05:00:18.652628+0000","last_peered":"2024-05-09T05:00:18.652628+0000","last_clean":"2024-05-09T05:00:18.652628+0000","last_became_active":"2024-05-09T05:00:16.880274+0000","last_became_peered":"2024-05-09T05:00:16.880274+0000","last_unstale":"2024-05-09T05:00:18.652628+0000","last_undegraded":"2024-05-09T05:00:18.652628+0000","last_fullsized":"2024-05-09T05:00:18.652628+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:27:18.295238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623650+0000","last_change":"2024-05-09T05:00:12.854427+0000","last_active":"2024-05-09T05:00:19.623650+0000","last_peered":"2024-05-09T05:00:19.623650+0000","last_clean":"2024-05-09T05:00:19.623650+0000","last_became_active":"2024-05-09T05:00:12.854089+0000","last_became_peered":"2024-05-09T05:00:12.854089+0000","last_unstale":"2024-05-09T05:00:19.623650+0000","last_undegraded":"2024-05-09T05:00:19.623650+0000","last_fullsized":"2024-05-09T05:00:19.623650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:33:12.899897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623926+0000","last_change":"2024-05-09T05:00:11.845606+0000","last_active":"2024-05-09T05:00:19.623926+0000","last_peered":"2024-05-09T05:00:19.623926+0000","last_clean":"2024-05-09T05:00:19.623926+0000","last_became_active":"2024-05-09T05:00:11.845227+0000","last_became_peered":"2024-05-09T05:00:11.845227+0000","last_unstale":"2024-05-09T05:00:19.623926+0000","last_undegraded":"2024-05-09T05:00:19.623926+0000","last_fullsized":"2024-05-09T05:00:19.623926+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:50.824636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970860+0000","last_change":"2024-05-09T05:00:16.880664+0000","last_active":"2024-05-09T05:00:19.970860+0000","last_peered":"2024-05-09T05:00:19.970860+0000","last_clean":"2024-05-09T05:00:19.970860+0000","last_became_active":"2024-05-09T05:00:16.880226+0000","last_became_peered":"2024-05-09T05:00:16.880226+0000","last_unstale":"2024-05-09T05:00:19.970860+0000","last_undegraded":"2024-05-09T05:00:19.970860+0000","last_fullsized":"2024-05-09T05:00:19.970860+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:41.016707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271152+0000","last_change":"2024-05-09T05:00:12.858289+0000","last_active":"2024-05-09T05:00:20.271152+0000","last_peered":"2024-05-09T05:00:20.271152+0000","last_clean":"2024-05-09T05:00:20.271152+0000","last_became_active":"2024-05-09T05:00:12.858013+0000","last_became_peered":"2024-05-09T05:00:12.858013+0000","last_unstale":"2024-05-09T05:00:20.271152+0000","last_undegraded":"2024-05-09T05:00:20.271152+0000","last_fullsized":"2024-05-09T05:00:20.271152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:50.968380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:22.574608+0000","last_change":"2024-05-09T05:00:11.842419+0000","last_active":"2024-05-09T05:00:22.574608+0000","last_peered":"2024-05-09T05:00:22.574608+0000","last_clean":"2024-05-09T05:00:22.574608+0000","last_became_active":"2024-05-09T05:00:11.842112+0000","last_became_peered":"2024-05-09T05:00:11.842112+0000","last_unstale":"2024-05-09T05:00:22.574608+0000","last_undegraded":"2024-05-09T05:00:22.574608+0000","last_fullsized":"2024-05-09T05:00:22.574608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:25.803010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271066+0000","last_change":"2024-05-09T05:00:16.874021+0000","last_active":"2024-05-09T05:00:20.271066+0000","last_peered":"2024-05-09T05:00:20.271066+0000","last_clean":"2024-05-09T05:00:20.271066+0000","last_became_active":"2024-05-09T05:00:16.873587+0000","last_became_peered":"2024-05-09T05:00:16.873587+0000","last_unstale":"2024-05-09T05:00:20.271066+0000","last_undegraded":"2024-05-09T05:00:20.271066+0000","last_fullsized":"2024-05-09T05:00:20.271066+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:47:02.206708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882366+0000","last_change":"2024-05-09T05:00:12.844150+0000","last_active":"2024-05-09T05:00:19.882366+0000","last_peered":"2024-05-09T05:00:19.882366+0000","last_clean":"2024-05-09T05:00:19.882366+0000","last_became_active":"2024-05-09T05:00:12.843845+0000","last_became_peered":"2024-05-09T05:00:12.843845+0000","last_unstale":"2024-05-09T05:00:19.882366+0000","last_undegraded":"2024-05-09T05:00:19.882366+0000","last_fullsized":"2024-05-09T05:00:19.882366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:27:58.422255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270961+0000","last_change":"2024-05-09T05:00:11.842387+0000","last_active":"2024-05-09T05:00:20.270961+0000","last_peered":"2024-05-09T05:00:20.270961+0000","last_clean":"2024-05-09T05:00:20.270961+0000","last_became_active":"2024-05-09T05:00:11.840749+0000","last_became_peered":"2024-05-09T05:00:11.840749+0000","last_unstale":"2024-05-09T05:00:20.270961+0000","last_undegraded":"2024-05-09T05:00:20.270961+0000","last_fullsized":"2024-05-09T05:00:20.270961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:27:57.420062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882161+0000","last_change":"2024-05-09T05:00:16.877382+0000","last_active":"2024-05-09T05:00:19.882161+0000","last_peered":"2024-05-09T05:00:19.882161+0000","last_clean":"2024-05-09T05:00:19.882161+0000","last_became_active":"2024-05-09T05:00:16.876966+0000","last_became_peered":"2024-05-09T05:00:16.876966+0000","last_unstale":"2024-05-09T05:00:19.882161+0000","last_undegraded":"2024-05-09T05:00:19.882161+0000","last_fullsized":"2024-05-09T05:00:19.882161+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:20:41.130506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624095+0000","last_change":"2024-05-09T05:00:12.844211+0000","last_active":"2024-05-09T05:00:19.624095+0000","last_peered":"2024-05-09T05:00:19.624095+0000","last_clean":"2024-05-09T05:00:19.624095+0000","last_became_active":"2024-05-09T05:00:12.843545+0000","last_became_peered":"2024-05-09T05:00:12.843545+0000","last_unstale":"2024-05-09T05:00:19.624095+0000","last_undegraded":"2024-05-09T05:00:19.624095+0000","last_fullsized":"2024-05-09T05:00:19.624095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970032+0000","last_change":"2024-05-09T05:00:11.841156+0000","last_active":"2024-05-09T05:00:19.970032+0000","last_peered":"2024-05-09T05:00:19.970032+0000","last_clean":"2024-05-09T05:00:19.970032+0000","last_became_active":"2024-05-09T05:00:11.840620+0000","last_became_peered":"2024-05-09T05:00:11.840620+0000","last_unstale":"2024-05-09T05:00:19.970032+0000","last_undegraded":"2024-05-09T05:00:19.970032+0000","last_fullsized":"2024-05-09T05:00:19.970032+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:10.779395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652218+0000","last_change":"2024-05-09T05:00:16.875531+0000","last_active":"2024-05-09T05:00:18.652218+0000","last_peered":"2024-05-09T05:00:18.652218+0000","last_clean":"2024-05-09T05:00:18.652218+0000","last_became_active":"2024-05-09T05:00:16.875056+0000","last_became_peered":"2024-05-09T05:00:16.875056+0000","last_unstale":"2024-05-09T05:00:18.652218+0000","last_undegraded":"2024-05-09T05:00:18.652218+0000","last_fullsized":"2024-05-09T05:00:18.652218+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:41.016707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970758+0000","last_change":"2024-05-09T05:00:12.852295+0000","last_active":"2024-05-09T05:00:19.970758+0000","last_peered":"2024-05-09T05:00:19.970758+0000","last_clean":"2024-05-09T05:00:19.970758+0000","last_became_active":"2024-05-09T05:00:12.851993+0000","last_became_peered":"2024-05-09T05:00:12.851993+0000","last_unstale":"2024-05-09T05:00:19.970758+0000","last_undegraded":"2024-05-09T05:00:19.970758+0000","last_fullsized":"2024-05-09T05:00:19.970758+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:44.359533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860982+0000","last_change":"2024-05-09T05:00:11.840063+0000","last_active":"2024-05-09T05:00:17.860982+0000","last_peered":"2024-05-09T05:00:17.860982+0000","last_clean":"2024-05-09T05:00:17.860982+0000","last_became_active":"2024-05-09T05:00:11.839772+0000","last_became_peered":"2024-05-09T05:00:11.839772+0000","last_unstale":"2024-05-09T05:00:17.860982+0000","last_undegraded":"2024-05-09T05:00:17.860982+0000","last_fullsized":"2024-05-09T05:00:17.860982+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:01.738771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652211+0000","last_change":"2024-05-09T05:00:16.883403+0000","last_active":"2024-05-09T05:00:18.652211+0000","last_peered":"2024-05-09T05:00:18.652211+0000","last_clean":"2024-05-09T05:00:18.652211+0000","last_became_active":"2024-05-09T05:00:16.882440+0000","last_became_peered":"2024-05-09T05:00:16.882440+0000","last_unstale":"2024-05-09T05:00:18.652211+0000","last_undegraded":"2024-05-09T05:00:18.652211+0000","last_fullsized":"2024-05-09T05:00:18.652211+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:18:15.413800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.624058+0000","last_change":"2024-05-09T05:00:16.883006+0000","last_active":"2024-05-09T05:00:19.624058+0000","last_peered":"2024-05-09T05:00:19.624058+0000","last_clean":"2024-05-09T05:00:19.624058+0000","last_became_active":"2024-05-09T05:00:16.882624+0000","last_became_peered":"2024-05-09T05:00:16.882624+0000","last_unstale":"2024-05-09T05:00:19.624058+0000","last_undegraded":"2024-05-09T05:00:19.624058+0000","last_fullsized":"2024-05-09T05:00:19.624058+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:02.171370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270485+0000","last_change":"2024-05-09T05:00:12.850411+0000","last_active":"2024-05-09T05:00:20.270485+0000","last_peered":"2024-05-09T05:00:20.270485+0000","last_clean":"2024-05-09T05:00:20.270485+0000","last_became_active":"2024-05-09T05:00:12.850121+0000","last_became_peered":"2024-05-09T05:00:12.850121+0000","last_unstale":"2024-05-09T05:00:20.270485+0000","last_undegraded":"2024-05-09T05:00:20.270485+0000","last_fullsized":"2024-05-09T05:00:20.270485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:30.691332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970418+0000","last_change":"2024-05-09T05:00:11.843746+0000","last_active":"2024-05-09T05:00:19.970418+0000","last_peered":"2024-05-09T05:00:19.970418+0000","last_clean":"2024-05-09T05:00:19.970418+0000","last_became_active":"2024-05-09T05:00:11.843423+0000","last_became_peered":"2024-05-09T05:00:11.843423+0000","last_unstale":"2024-05-09T05:00:19.970418+0000","last_undegraded":"2024-05-09T05:00:19.970418+0000","last_fullsized":"2024-05-09T05:00:19.970418+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:41:27.364045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860975+0000","last_change":"2024-05-09T05:00:16.873412+0000","last_active":"2024-05-09T05:00:17.860975+0000","last_peered":"2024-05-09T05:00:17.860975+0000","last_clean":"2024-05-09T05:00:17.860975+0000","last_became_active":"2024-05-09T05:00:16.873014+0000","last_became_peered":"2024-05-09T05:00:16.873014+0000","last_unstale":"2024-05-09T05:00:17.860975+0000","last_undegraded":"2024-05-09T05:00:17.860975+0000","last_fullsized":"2024-05-09T05:00:17.860975+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:20.415639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652208+0000","last_change":"2024-05-09T05:00:12.856243+0000","last_active":"2024-05-09T05:00:18.652208+0000","last_peered":"2024-05-09T05:00:18.652208+0000","last_clean":"2024-05-09T05:00:18.652208+0000","last_became_active":"2024-05-09T05:00:12.854306+0000","last_became_peered":"2024-05-09T05:00:12.854306+0000","last_unstale":"2024-05-09T05:00:18.652208+0000","last_undegraded":"2024-05-09T05:00:18.652208+0000","last_fullsized":"2024-05-09T05:00:18.652208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:10:10.149613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652448+0000","last_change":"2024-05-09T05:00:11.837046+0000","last_active":"2024-05-09T05:00:18.652448+0000","last_peered":"2024-05-09T05:00:18.652448+0000","last_clean":"2024-05-09T05:00:18.652448+0000","last_became_active":"2024-05-09T05:00:11.836773+0000","last_became_peered":"2024-05-09T05:00:11.836773+0000","last_unstale":"2024-05-09T05:00:18.652448+0000","last_undegraded":"2024-05-09T05:00:18.652448+0000","last_fullsized":"2024-05-09T05:00:18.652448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:48:26.640184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860385+0000","last_change":"2024-05-09T05:00:11.841442+0000","last_active":"2024-05-09T05:00:17.860385+0000","last_peered":"2024-05-09T05:00:17.860385+0000","last_clean":"2024-05-09T05:00:17.860385+0000","last_became_active":"2024-05-09T05:00:11.841099+0000","last_became_peered":"2024-05-09T05:00:11.841099+0000","last_unstale":"2024-05-09T05:00:17.860385+0000","last_undegraded":"2024-05-09T05:00:17.860385+0000","last_fullsized":"2024-05-09T05:00:17.860385+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:41:27.364045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623878+0000","last_change":"2024-05-09T05:00:12.845049+0000","last_active":"2024-05-09T05:00:19.623878+0000","last_peered":"2024-05-09T05:00:19.623878+0000","last_clean":"2024-05-09T05:00:19.623878+0000","last_became_active":"2024-05-09T05:00:12.844724+0000","last_became_peered":"2024-05-09T05:00:12.844724+0000","last_unstale":"2024-05-09T05:00:19.623878+0000","last_undegraded":"2024-05-09T05:00:19.623878+0000","last_fullsized":"2024-05-09T05:00:19.623878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:45.463031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860933+0000","last_change":"2024-05-09T05:00:16.883497+0000","last_active":"2024-05-09T05:00:17.860933+0000","last_peered":"2024-05-09T05:00:17.860933+0000","last_clean":"2024-05-09T05:00:17.860933+0000","last_became_active":"2024-05-09T05:00:16.883063+0000","last_became_peered":"2024-05-09T05:00:16.883063+0000","last_unstale":"2024-05-09T05:00:17.860933+0000","last_undegraded":"2024-05-09T05:00:17.860933+0000","last_fullsized":"2024-05-09T05:00:17.860933+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:04:58.832049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652723+0000","last_change":"2024-05-09T05:00:11.847505+0000","last_active":"2024-05-09T05:00:18.652723+0000","last_peered":"2024-05-09T05:00:18.652723+0000","last_clean":"2024-05-09T05:00:18.652723+0000","last_became_active":"2024-05-09T05:00:11.847187+0000","last_became_peered":"2024-05-09T05:00:11.847187+0000","last_unstale":"2024-05-09T05:00:18.652723+0000","last_undegraded":"2024-05-09T05:00:18.652723+0000","last_fullsized":"2024-05-09T05:00:18.652723+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:10.814732+0000","last_clean_scrub_stamp":"2024-05-09T05:00:10.814732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:50.824636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882122+0000","last_change":"2024-05-09T05:00:12.838998+0000","last_active":"2024-05-09T05:00:19.882122+0000","last_peered":"2024-05-09T05:00:19.882122+0000","last_clean":"2024-05-09T05:00:19.882122+0000","last_became_active":"2024-05-09T05:00:12.838702+0000","last_became_peered":"2024-05-09T05:00:12.838702+0000","last_unstale":"2024-05-09T05:00:19.882122+0000","last_undegraded":"2024-05-09T05:00:19.882122+0000","last_fullsized":"2024-05-09T05:00:19.882122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:50.662283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652939+0000","last_change":"2024-05-09T05:00:16.876805+0000","last_active":"2024-05-09T05:00:18.652939+0000","last_peered":"2024-05-09T05:00:18.652939+0000","last_clean":"2024-05-09T05:00:18.652939+0000","last_became_active":"2024-05-09T05:00:16.876265+0000","last_became_peered":"2024-05-09T05:00:16.876265+0000","last_unstale":"2024-05-09T05:00:18.652939+0000","last_undegraded":"2024-05-09T05:00:18.652939+0000","last_fullsized":"2024-05-09T05:00:18.652939+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:15.845002+0000","last_clean_scrub_stamp":"2024-05-09T05:00:15.845002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:54:27.622037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860099+0000","last_change":"2024-05-09T05:00:12.857527+0000","last_active":"2024-05-09T05:00:17.860099+0000","last_peered":"2024-05-09T05:00:17.860099+0000","last_clean":"2024-05-09T05:00:17.860099+0000","last_became_active":"2024-05-09T05:00:12.857202+0000","last_became_peered":"2024-05-09T05:00:12.857202+0000","last_unstale":"2024-05-09T05:00:17.860099+0000","last_undegraded":"2024-05-09T05:00:17.860099+0000","last_fullsized":"2024-05-09T05:00:17.860099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:45.463031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860631+0000","last_change":"2024-05-09T05:00:12.844334+0000","last_active":"2024-05-09T05:00:17.860631+0000","last_peered":"2024-05-09T05:00:17.860631+0000","last_clean":"2024-05-09T05:00:17.860631+0000","last_became_active":"2024-05-09T05:00:12.844018+0000","last_became_peered":"2024-05-09T05:00:12.844018+0000","last_unstale":"2024-05-09T05:00:17.860631+0000","last_undegraded":"2024-05-09T05:00:17.860631+0000","last_fullsized":"2024-05-09T05:00:17.860631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:40.797004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970591+0000","last_change":"2024-05-09T05:00:12.855985+0000","last_active":"2024-05-09T05:00:19.970591+0000","last_peered":"2024-05-09T05:00:19.970591+0000","last_clean":"2024-05-09T05:00:19.970591+0000","last_became_active":"2024-05-09T05:00:12.855586+0000","last_became_peered":"2024-05-09T05:00:12.855586+0000","last_unstale":"2024-05-09T05:00:19.970591+0000","last_undegraded":"2024-05-09T05:00:19.970591+0000","last_fullsized":"2024-05-09T05:00:19.970591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:06.793812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623781+0000","last_change":"2024-05-09T05:00:12.848323+0000","last_active":"2024-05-09T05:00:19.623781+0000","last_peered":"2024-05-09T05:00:19.623781+0000","last_clean":"2024-05-09T05:00:19.623781+0000","last_became_active":"2024-05-09T05:00:12.848053+0000","last_became_peered":"2024-05-09T05:00:12.848053+0000","last_unstale":"2024-05-09T05:00:19.623781+0000","last_undegraded":"2024-05-09T05:00:19.623781+0000","last_fullsized":"2024-05-09T05:00:19.623781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:19.080163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652749+0000","last_change":"2024-05-09T05:00:12.849804+0000","last_active":"2024-05-09T05:00:18.652749+0000","last_peered":"2024-05-09T05:00:18.652749+0000","last_clean":"2024-05-09T05:00:18.652749+0000","last_became_active":"2024-05-09T05:00:12.849530+0000","last_became_peered":"2024-05-09T05:00:12.849530+0000","last_unstale":"2024-05-09T05:00:18.652749+0000","last_undegraded":"2024-05-09T05:00:18.652749+0000","last_fullsized":"2024-05-09T05:00:18.652749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:20:37.102430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270858+0000","last_change":"2024-05-09T05:00:12.848438+0000","last_active":"2024-05-09T05:00:20.270858+0000","last_peered":"2024-05-09T05:00:20.270858+0000","last_clean":"2024-05-09T05:00:20.270858+0000","last_became_active":"2024-05-09T05:00:12.848176+0000","last_became_peered":"2024-05-09T05:00:12.848176+0000","last_unstale":"2024-05-09T05:00:20.270858+0000","last_undegraded":"2024-05-09T05:00:20.270858+0000","last_fullsized":"2024-05-09T05:00:20.270858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860997+0000","last_change":"2024-05-09T05:00:12.855310+0000","last_active":"2024-05-09T05:00:17.860997+0000","last_peered":"2024-05-09T05:00:17.860997+0000","last_clean":"2024-05-09T05:00:17.860997+0000","last_became_active":"2024-05-09T05:00:12.854031+0000","last_became_peered":"2024-05-09T05:00:12.854031+0000","last_unstale":"2024-05-09T05:00:17.860997+0000","last_undegraded":"2024-05-09T05:00:17.860997+0000","last_fullsized":"2024-05-09T05:00:17.860997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:44.359533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652800+0000","last_change":"2024-05-09T05:00:12.849125+0000","last_active":"2024-05-09T05:00:18.652800+0000","last_peered":"2024-05-09T05:00:18.652800+0000","last_clean":"2024-05-09T05:00:18.652800+0000","last_became_active":"2024-05-09T05:00:12.848852+0000","last_became_peered":"2024-05-09T05:00:12.848852+0000","last_unstale":"2024-05-09T05:00:18.652800+0000","last_undegraded":"2024-05-09T05:00:18.652800+0000","last_fullsized":"2024-05-09T05:00:18.652800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970301+0000","last_change":"2024-05-09T05:00:12.847892+0000","last_active":"2024-05-09T05:00:19.970301+0000","last_peered":"2024-05-09T05:00:19.970301+0000","last_clean":"2024-05-09T05:00:19.970301+0000","last_became_active":"2024-05-09T05:00:12.847051+0000","last_became_peered":"2024-05-09T05:00:12.847051+0000","last_unstale":"2024-05-09T05:00:19.970301+0000","last_undegraded":"2024-05-09T05:00:19.970301+0000","last_fullsized":"2024-05-09T05:00:19.970301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652323+0000","last_change":"2024-05-09T05:00:12.848463+0000","last_active":"2024-05-09T05:00:18.652323+0000","last_peered":"2024-05-09T05:00:18.652323+0000","last_clean":"2024-05-09T05:00:18.652323+0000","last_became_active":"2024-05-09T05:00:12.848170+0000","last_became_peered":"2024-05-09T05:00:12.848170+0000","last_unstale":"2024-05-09T05:00:18.652323+0000","last_undegraded":"2024-05-09T05:00:18.652323+0000","last_fullsized":"2024-05-09T05:00:18.652323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:45.463031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970492+0000","last_change":"2024-05-09T05:00:12.852750+0000","last_active":"2024-05-09T05:00:19.970492+0000","last_peered":"2024-05-09T05:00:19.970492+0000","last_clean":"2024-05-09T05:00:19.970492+0000","last_became_active":"2024-05-09T05:00:12.852487+0000","last_became_peered":"2024-05-09T05:00:12.852487+0000","last_unstale":"2024-05-09T05:00:19.970492+0000","last_undegraded":"2024-05-09T05:00:19.970492+0000","last_fullsized":"2024-05-09T05:00:19.970492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:07:50.968380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623652+0000","last_change":"2024-05-09T05:00:12.850281+0000","last_active":"2024-05-09T05:00:19.623652+0000","last_peered":"2024-05-09T05:00:19.623652+0000","last_clean":"2024-05-09T05:00:19.623652+0000","last_became_active":"2024-05-09T05:00:12.849875+0000","last_became_peered":"2024-05-09T05:00:12.849875+0000","last_unstale":"2024-05-09T05:00:19.623652+0000","last_undegraded":"2024-05-09T05:00:19.623652+0000","last_fullsized":"2024-05-09T05:00:19.623652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:10:10.149613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860536+0000","last_change":"2024-05-09T05:00:12.841388+0000","last_active":"2024-05-09T05:00:17.860536+0000","last_peered":"2024-05-09T05:00:17.860536+0000","last_clean":"2024-05-09T05:00:17.860536+0000","last_became_active":"2024-05-09T05:00:12.841121+0000","last_became_peered":"2024-05-09T05:00:12.841121+0000","last_unstale":"2024-05-09T05:00:17.860536+0000","last_undegraded":"2024-05-09T05:00:17.860536+0000","last_fullsized":"2024-05-09T05:00:17.860536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:30.691332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271354+0000","last_change":"2024-05-09T05:00:12.853880+0000","last_active":"2024-05-09T05:00:20.271354+0000","last_peered":"2024-05-09T05:00:20.271354+0000","last_clean":"2024-05-09T05:00:20.271354+0000","last_became_active":"2024-05-09T05:00:12.852044+0000","last_became_peered":"2024-05-09T05:00:12.852044+0000","last_unstale":"2024-05-09T05:00:20.271354+0000","last_undegraded":"2024-05-09T05:00:20.271354+0000","last_fullsized":"2024-05-09T05:00:20.271354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:31:56.397467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860942+0000","last_change":"2024-05-09T05:00:12.842477+0000","last_active":"2024-05-09T05:00:17.860942+0000","last_peered":"2024-05-09T05:00:17.860942+0000","last_clean":"2024-05-09T05:00:17.860942+0000","last_became_active":"2024-05-09T05:00:12.842135+0000","last_became_peered":"2024-05-09T05:00:12.842135+0000","last_unstale":"2024-05-09T05:00:17.860942+0000","last_undegraded":"2024-05-09T05:00:17.860942+0000","last_fullsized":"2024-05-09T05:00:17.860942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:27:58.422255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.971029+0000","last_change":"2024-05-09T05:00:12.852559+0000","last_active":"2024-05-09T05:00:19.971029+0000","last_peered":"2024-05-09T05:00:19.971029+0000","last_clean":"2024-05-09T05:00:19.971029+0000","last_became_active":"2024-05-09T05:00:12.852370+0000","last_became_peered":"2024-05-09T05:00:12.852370+0000","last_unstale":"2024-05-09T05:00:19.971029+0000","last_undegraded":"2024-05-09T05:00:19.971029+0000","last_fullsized":"2024-05-09T05:00:19.971029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:58.178632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970409+0000","last_change":"2024-05-09T05:00:12.856404+0000","last_active":"2024-05-09T05:00:19.970409+0000","last_peered":"2024-05-09T05:00:19.970409+0000","last_clean":"2024-05-09T05:00:19.970409+0000","last_became_active":"2024-05-09T05:00:12.856126+0000","last_became_peered":"2024-05-09T05:00:12.856126+0000","last_unstale":"2024-05-09T05:00:19.970409+0000","last_undegraded":"2024-05-09T05:00:19.970409+0000","last_fullsized":"2024-05-09T05:00:19.970409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270574+0000","last_change":"2024-05-09T05:00:12.847697+0000","last_active":"2024-05-09T05:00:20.270574+0000","last_peered":"2024-05-09T05:00:20.270574+0000","last_clean":"2024-05-09T05:00:20.270574+0000","last_became_active":"2024-05-09T05:00:12.847412+0000","last_became_peered":"2024-05-09T05:00:12.847412+0000","last_unstale":"2024-05-09T05:00:20.270574+0000","last_undegraded":"2024-05-09T05:00:20.270574+0000","last_fullsized":"2024-05-09T05:00:20.270574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:19.080163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623651+0000","last_change":"2024-05-09T05:00:12.846578+0000","last_active":"2024-05-09T05:00:19.623651+0000","last_peered":"2024-05-09T05:00:19.623651+0000","last_clean":"2024-05-09T05:00:19.623651+0000","last_became_active":"2024-05-09T05:00:12.844465+0000","last_became_peered":"2024-05-09T05:00:12.844465+0000","last_unstale":"2024-05-09T05:00:19.623651+0000","last_undegraded":"2024-05-09T05:00:19.623651+0000","last_fullsized":"2024-05-09T05:00:19.623651+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:17:11.682946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652503+0000","last_change":"2024-05-09T05:00:12.852179+0000","last_active":"2024-05-09T05:00:18.652503+0000","last_peered":"2024-05-09T05:00:18.652503+0000","last_clean":"2024-05-09T05:00:18.652503+0000","last_became_active":"2024-05-09T05:00:12.849715+0000","last_became_peered":"2024-05-09T05:00:12.849715+0000","last_unstale":"2024-05-09T05:00:18.652503+0000","last_undegraded":"2024-05-09T05:00:18.652503+0000","last_fullsized":"2024-05-09T05:00:18.652503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:19:19.080163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.882233+0000","last_change":"2024-05-09T05:00:12.837696+0000","last_active":"2024-05-09T05:00:19.882233+0000","last_peered":"2024-05-09T05:00:19.882233+0000","last_clean":"2024-05-09T05:00:19.882233+0000","last_became_active":"2024-05-09T05:00:12.837445+0000","last_became_peered":"2024-05-09T05:00:12.837445+0000","last_unstale":"2024-05-09T05:00:19.882233+0000","last_undegraded":"2024-05-09T05:00:19.882233+0000","last_fullsized":"2024-05-09T05:00:19.882233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:40.797004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860739+0000","last_change":"2024-05-09T05:00:12.840693+0000","last_active":"2024-05-09T05:00:17.860739+0000","last_peered":"2024-05-09T05:00:17.860739+0000","last_clean":"2024-05-09T05:00:17.860739+0000","last_became_active":"2024-05-09T05:00:12.840400+0000","last_became_peered":"2024-05-09T05:00:12.840400+0000","last_unstale":"2024-05-09T05:00:17.860739+0000","last_undegraded":"2024-05-09T05:00:17.860739+0000","last_fullsized":"2024-05-09T05:00:17.860739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:24.896256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:17.860995+0000","last_change":"2024-05-09T05:00:12.841318+0000","last_active":"2024-05-09T05:00:17.860995+0000","last_peered":"2024-05-09T05:00:17.860995+0000","last_clean":"2024-05-09T05:00:17.860995+0000","last_became_active":"2024-05-09T05:00:12.841026+0000","last_became_peered":"2024-05-09T05:00:12.841026+0000","last_unstale":"2024-05-09T05:00:17.860995+0000","last_undegraded":"2024-05-09T05:00:17.860995+0000","last_fullsized":"2024-05-09T05:00:17.860995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271552+0000","last_change":"2024-05-09T05:00:12.852840+0000","last_active":"2024-05-09T05:00:20.271552+0000","last_peered":"2024-05-09T05:00:20.271552+0000","last_clean":"2024-05-09T05:00:20.271552+0000","last_became_active":"2024-05-09T05:00:12.852259+0000","last_became_peered":"2024-05-09T05:00:12.852259+0000","last_unstale":"2024-05-09T05:00:20.271552+0000","last_undegraded":"2024-05-09T05:00:20.271552+0000","last_fullsized":"2024-05-09T05:00:20.271552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:16:58.143294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:18.652120+0000","last_change":"2024-05-09T05:00:12.846320+0000","last_active":"2024-05-09T05:00:18.652120+0000","last_peered":"2024-05-09T05:00:18.652120+0000","last_clean":"2024-05-09T05:00:18.652120+0000","last_became_active":"2024-05-09T05:00:12.845732+0000","last_became_peered":"2024-05-09T05:00:12.845732+0000","last_unstale":"2024-05-09T05:00:18.652120+0000","last_undegraded":"2024-05-09T05:00:18.652120+0000","last_fullsized":"2024-05-09T05:00:18.652120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:33:12.899897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270936+0000","last_change":"2024-05-09T05:00:12.858255+0000","last_active":"2024-05-09T05:00:20.270936+0000","last_peered":"2024-05-09T05:00:20.270936+0000","last_clean":"2024-05-09T05:00:20.270936+0000","last_became_active":"2024-05-09T05:00:12.857971+0000","last_became_peered":"2024-05-09T05:00:12.857971+0000","last_unstale":"2024-05-09T05:00:20.270936+0000","last_undegraded":"2024-05-09T05:00:20.270936+0000","last_fullsized":"2024-05-09T05:00:20.270936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:23.066586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970422+0000","last_change":"2024-05-09T05:00:12.846491+0000","last_active":"2024-05-09T05:00:19.970422+0000","last_peered":"2024-05-09T05:00:19.970422+0000","last_clean":"2024-05-09T05:00:19.970422+0000","last_became_active":"2024-05-09T05:00:12.845643+0000","last_became_peered":"2024-05-09T05:00:12.845643+0000","last_unstale":"2024-05-09T05:00:19.970422+0000","last_undegraded":"2024-05-09T05:00:19.970422+0000","last_fullsized":"2024-05-09T05:00:19.970422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:20:37.102430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.270963+0000","last_change":"2024-05-09T05:00:12.854820+0000","last_active":"2024-05-09T05:00:20.270963+0000","last_peered":"2024-05-09T05:00:20.270963+0000","last_clean":"2024-05-09T05:00:20.270963+0000","last_became_active":"2024-05-09T05:00:12.853663+0000","last_became_peered":"2024-05-09T05:00:12.853663+0000","last_unstale":"2024-05-09T05:00:20.270963+0000","last_undegraded":"2024-05-09T05:00:20.270963+0000","last_fullsized":"2024-05-09T05:00:20.270963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:50:04.977127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.970662+0000","last_change":"2024-05-09T05:00:12.855723+0000","last_active":"2024-05-09T05:00:19.970662+0000","last_peered":"2024-05-09T05:00:19.970662+0000","last_clean":"2024-05-09T05:00:19.970662+0000","last_became_active":"2024-05-09T05:00:12.855479+0000","last_became_peered":"2024-05-09T05:00:12.855479+0000","last_unstale":"2024-05-09T05:00:19.970662+0000","last_undegraded":"2024-05-09T05:00:19.970662+0000","last_fullsized":"2024-05-09T05:00:19.970662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:01:08.488591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:19.623805+0000","last_change":"2024-05-09T05:00:12.838168+0000","last_active":"2024-05-09T05:00:19.623805+0000","last_peered":"2024-05-09T05:00:19.623805+0000","last_clean":"2024-05-09T05:00:19.623805+0000","last_became_active":"2024-05-09T05:00:12.837892+0000","last_became_peered":"2024-05-09T05:00:12.837892+0000","last_unstale":"2024-05-09T05:00:19.623805+0000","last_undegraded":"2024-05-09T05:00:19.623805+0000","last_fullsized":"2024-05-09T05:00:19.623805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:30.691332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271359+0000","last_change":"2024-05-09T05:00:12.851393+0000","last_active":"2024-05-09T05:00:20.271359+0000","last_peered":"2024-05-09T05:00:20.271359+0000","last_clean":"2024-05-09T05:00:20.271359+0000","last_became_active":"2024-05-09T05:00:12.851057+0000","last_became_peered":"2024-05-09T05:00:12.851057+0000","last_unstale":"2024-05-09T05:00:20.271359+0000","last_undegraded":"2024-05-09T05:00:20.271359+0000","last_fullsized":"2024-05-09T05:00:20.271359+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:33:30.386024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-05-09T05:00:20.271442+0000","last_change":"2024-05-09T05:00:12.853084+0000","last_active":"2024-05-09T05:00:20.271442+0000","last_peered":"2024-05-09T05:00:20.271442+0000","last_clean":"2024-05-09T05:00:20.271442+0000","last_became_active":"2024-05-09T05:00:12.852602+0000","last_became_peered":"2024-05-09T05:00:12.852602+0000","last_unstale":"2024-05-09T05:00:20.271442+0000","last_undegraded":"2024-05-09T05:00:20.271442+0000","last_fullsized":"2024-05-09T05:00:20.271442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:11.816926+0000","last_clean_scrub_stamp":"2024-05-09T05:00:11.816926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:58.178632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:00:36.287 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-09T05:00:36.287 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-09T05:00:36.287 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-05-09T05:00:36.287 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-09T05:00:36.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.424+0000 7fa6f5189640 1 Processor -- start 2024-05-09T05:00:36.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.424+0000 7fa6f5189640 1 -- start start 2024-05-09T05:00:36.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.425+0000 7fa6f5189640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa6f0097de0 0x7fa6f0099b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:36.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.425+0000 7fa6f5189640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f015c430 con 0x7fa6f0072c00 2024-05-09T05:00:36.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.425+0000 7fa6f5189640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f009b470 con 0x7fa6f0070150 2024-05-09T05:00:36.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.425+0000 7fa6f5189640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fa6f009b220 con 0x7fa6f0097de0 2024-05-09T05:00:36.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.425+0000 7fa6ef59e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa6f0072c00 0x7fa6f009a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38516/0 (socket says 172.21.15.60:38516) 2024-05-09T05:00:36.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.425+0000 7fa6ef59e640 1 -- 172.21.15.60:0/4281418965 learned_addr learned my addr 172.21.15.60:0/4281418965 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:36.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.425+0000 7fa6ee59c640 1 --2- 172.21.15.60:0/4281418965 >> v2:172.21.15.114:3300/0 conn(0x7fa6f0097de0 0x7fa6f0099b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:36.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.425+0000 7fa6edd9b640 1 -- 172.21.15.60:0/4281418965 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 204035512 0 0) 0x7fa6f015c430 con 0x7fa6f0072c00 2024-05-09T05:00:36.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.425+0000 7fa6edd9b640 1 -- 172.21.15.60:0/4281418965 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6dc003520 con 0x7fa6f0072c00 2024-05-09T05:00:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.426+0000 7fa6ee59c640 1 -- 172.21.15.60:0/4281418965 >> v1:172.21.15.114:6789/0 conn(0x7fa6f0070150 legacy=0x7fa6f0070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.426+0000 7fa6ee59c640 1 -- 172.21.15.60:0/4281418965 >> v1:172.21.15.60:6789/0 conn(0x7fa6f0072c00 legacy=0x7fa6f009a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.426+0000 7fa6ee59c640 1 -- 172.21.15.60:0/4281418965 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f006e950 con 0x7fa6f0097de0 2024-05-09T05:00:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.426+0000 7fa6ee59c640 1 --2- 172.21.15.60:0/4281418965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f0097de0 0x7fa6f0099b50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa6d80097a0 tx=0x7fa6d802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=761ff8f401a493f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.427+0000 7fa6edd9b640 1 -- 172.21.15.60:0/4281418965 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8037070 con 0x7fa6f0097de0 2024-05-09T05:00:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.427+0000 7fa6edd9b640 1 -- 172.21.15.60:0/4281418965 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6d8032440 con 0x7fa6f0097de0 2024-05-09T05:00:36.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.427+0000 7fa6edd9b640 1 -- 172.21.15.60:0/4281418965 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d80327f0 con 0x7fa6f0097de0 2024-05-09T05:00:36.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.427+0000 7fa6f5189640 1 -- 172.21.15.60:0/4281418965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f0097de0 msgr2=0x7fa6f0099b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:36.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.427+0000 7fa6f5189640 1 --2- 172.21.15.60:0/4281418965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f0097de0 0x7fa6f0099b50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa6d80097a0 tx=0x7fa6d802a230 comp rx=0 tx=0).stop 2024-05-09T05:00:36.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.427+0000 7fa6f5189640 1 -- 172.21.15.60:0/4281418965 shutdown_connections 2024-05-09T05:00:36.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.427+0000 7fa6f5189640 1 --2- 172.21.15.60:0/4281418965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f0097de0 0x7fa6f0099b50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.427+0000 7fa6f5189640 1 -- 172.21.15.60:0/4281418965 >> 172.21.15.60:0/4281418965 conn(0x7fa6f00972f0 msgr2=0x7fa6f00976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:36.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.427+0000 7fa6f5189640 1 -- 172.21.15.60:0/4281418965 shutdown_connections 2024-05-09T05:00:36.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.427+0000 7fa6f5189640 1 -- 172.21.15.60:0/4281418965 wait complete. 2024-05-09T05:00:36.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.429+0000 7fa6f5189640 1 Processor -- start 2024-05-09T05:00:36.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.429+0000 7fa6f5189640 1 -- start start 2024-05-09T05:00:36.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.429+0000 7fa6f5189640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f0070150 0x7fa6f00a8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:36.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.429+0000 7fa6f5189640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6f0072c00 0x7fa6f01f5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:36.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.429+0000 7fa6f5189640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6f0097de0 0x7fa6f01f5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:36.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.429+0000 7fa6f5189640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f009a7a0 con 0x7fa6f0072c00 2024-05-09T05:00:36.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.429+0000 7fa6f5189640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f00b46e0 con 0x7fa6f0070150 2024-05-09T05:00:36.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.429+0000 7fa6f5189640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa6f009a920 con 0x7fa6f0097de0 2024-05-09T05:00:36.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.429+0000 7fa6ee59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6f0072c00 0x7fa6f01f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:36.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.430+0000 7fa6ee59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6f0072c00 0x7fa6f01f5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35140/0 (socket says 172.21.15.60:35140) 2024-05-09T05:00:36.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.430+0000 7fa6ee59c640 1 -- 172.21.15.60:0/2255143925 learned_addr learned my addr 172.21.15.60:0/2255143925 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:36.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.430+0000 7fa6ef59e640 1 --2- 172.21.15.60:0/2255143925 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6f0097de0 0x7fa6f01f5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:36.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.430+0000 7fa6eed9d640 1 --2- 172.21.15.60:0/2255143925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f0070150 0x7fa6f00a8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:36.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.430+0000 7fa6ee59c640 1 -- 172.21.15.60:0/2255143925 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6f0097de0 msgr2=0x7fa6f01f5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:36.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.430+0000 7fa6ee59c640 1 --2- 172.21.15.60:0/2255143925 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6f0097de0 0x7fa6f01f5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.430+0000 7fa6ee59c640 1 -- 172.21.15.60:0/2255143925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f0070150 msgr2=0x7fa6f00a8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:36.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.430+0000 7fa6ee59c640 1 --2- 172.21.15.60:0/2255143925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f0070150 0x7fa6f00a8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.430+0000 7fa6ee59c640 1 -- 172.21.15.60:0/2255143925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f01e4d00 con 0x7fa6f0072c00 2024-05-09T05:00:36.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.430+0000 7fa6ee59c640 1 --2- 172.21.15.60:0/2255143925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6f0072c00 0x7fa6f01f5310 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8035040 tx=0x7fa6d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:36.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.430+0000 7fa6e77fe640 1 -- 172.21.15.60:0/2255143925 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8037050 con 0x7fa6f0072c00 2024-05-09T05:00:36.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.430+0000 7fa6e77fe640 1 -- 172.21.15.60:0/2255143925 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6d8038040 con 0x7fa6f0072c00 2024-05-09T05:00:36.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.430+0000 7fa6f5189640 1 -- 172.21.15.60:0/2255143925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6f01e4740 con 0x7fa6f0072c00 2024-05-09T05:00:36.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.430+0000 7fa6e77fe640 1 -- 172.21.15.60:0/2255143925 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8004450 con 0x7fa6f0072c00 2024-05-09T05:00:36.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.430+0000 7fa6f5189640 1 -- 172.21.15.60:0/2255143925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6f01f4df0 con 0x7fa6f0072c00 2024-05-09T05:00:36.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.431+0000 7fa6f5189640 1 -- 172.21.15.60:0/2255143925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6f0051040 con 0x7fa6f0072c00 2024-05-09T05:00:36.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.432+0000 7fa6e77fe640 1 -- 172.21.15.60:0/2255143925 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa6d803c6d0 con 0x7fa6f0072c00 2024-05-09T05:00:36.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.433+0000 7fa6e77fe640 1 --2- 172.21.15.60:0/2255143925 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa6d00762d0 0x7fa6d0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:36.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.433+0000 7fa6e77fe640 1 -- 172.21.15.60:0/2255143925 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa6d80b8330 con 0x7fa6f0072c00 2024-05-09T05:00:36.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.433+0000 7fa6eed9d640 1 --2- 172.21.15.60:0/2255143925 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa6d00762d0 0x7fa6d0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:36.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.433+0000 7fa6eed9d640 1 --2- 172.21.15.60:0/2255143925 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa6d00762d0 0x7fa6d0078790 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa6e00012e0 tx=0x7fa6e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:36.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.437+0000 7fa6e77fe640 1 -- 172.21.15.60:0/2255143925 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa6d8081260 con 0x7fa6f0072c00 2024-05-09T05:00:36.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.671+0000 7fa6f5189640 1 -- 172.21.15.60:0/2255143925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa6f015c430 con 0x7fa6f0072c00 2024-05-09T05:00:36.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.671+0000 7fa6e77fe640 1 -- 172.21.15.60:0/2255143925 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fa6d80b80f0 con 0x7fa6f0072c00 2024-05-09T05:00:36.673 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:36.673 INFO:teuthology.orchestra.run.smithi060.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-09T05:00:36.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.674+0000 7fa6f5189640 1 -- 172.21.15.60:0/2255143925 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa6d00762d0 msgr2=0x7fa6d0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:36.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.674+0000 7fa6f5189640 1 --2- 172.21.15.60:0/2255143925 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa6d00762d0 0x7fa6d0078790 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa6e00012e0 tx=0x7fa6e0009040 comp rx=0 tx=0).stop 2024-05-09T05:00:36.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.674+0000 7fa6f5189640 1 -- 172.21.15.60:0/2255143925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6f0072c00 msgr2=0x7fa6f01f5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:36.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.674+0000 7fa6f5189640 1 --2- 172.21.15.60:0/2255143925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6f0072c00 0x7fa6f01f5310 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8035040 tx=0x7fa6d80046c0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.674+0000 7fa6f5189640 1 -- 172.21.15.60:0/2255143925 shutdown_connections 2024-05-09T05:00:36.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.674+0000 7fa6f5189640 1 --2- 172.21.15.60:0/2255143925 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa6d00762d0 0x7fa6d0078790 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.674+0000 7fa6f5189640 1 --2- 172.21.15.60:0/2255143925 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa6f0097de0 0x7fa6f01f5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.674+0000 7fa6f5189640 1 --2- 172.21.15.60:0/2255143925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6f0072c00 0x7fa6f01f5310 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.674+0000 7fa6f5189640 1 --2- 172.21.15.60:0/2255143925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f0070150 0x7fa6f00a8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.674+0000 7fa6f5189640 1 -- 172.21.15.60:0/2255143925 >> 172.21.15.60:0/2255143925 conn(0x7fa6f00972f0 msgr2=0x7fa6f00b34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:36.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.674+0000 7fa6f5189640 1 -- 172.21.15.60:0/2255143925 shutdown_connections 2024-05-09T05:00:36.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.674+0000 7fa6f5189640 1 -- 172.21.15.60:0/2255143925 wait complete. 2024-05-09T05:00:36.685 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-05-09T05:00:36.686 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:00:36.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.790+0000 7f6f5f789640 1 Processor -- start 2024-05-09T05:00:36.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.790+0000 7f6f5f789640 1 -- start start 2024-05-09T05:00:36.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.791+0000 7f6f5f789640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6f5815f490 0x7f6f580ad010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:36.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.791+0000 7f6f5f789640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6f580737a0 0x7f6f580722f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:36.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.791+0000 7f6f5f789640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f580ad6f0 con 0x7f6f5806f0c0 2024-05-09T05:00:36.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.791+0000 7f6f5f789640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f6f580718a0 con 0x7f6f580737a0 2024-05-09T05:00:36.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.791+0000 7f6f5f789640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f6f580ad9d0 con 0x7f6f5815f490 2024-05-09T05:00:36.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.791+0000 7f6f5cd24640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6f580737a0 0x7f6f580722f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:36.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.791+0000 7f6f5dd26640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6f5806f0c0 0x7f6f580af070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38532/0 (socket says 172.21.15.60:38532) 2024-05-09T05:00:36.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.791+0000 7f6f5dd26640 1 -- 172.21.15.60:0/1080125047 learned_addr learned my addr 172.21.15.60:0/1080125047 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:36.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.791+0000 7f6f5d525640 1 --2- 172.21.15.60:0/1080125047 >> v2:172.21.15.114:3300/0 conn(0x7f6f5815f490 0x7f6f580ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:36.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.791+0000 7f6f5cd24640 1 -- 172.21.15.60:0/1080125047 >> v2:172.21.15.114:3300/0 conn(0x7f6f5815f490 msgr2=0x7f6f580ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:36.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.791+0000 7f6f5cd24640 1 --2- 172.21.15.60:0/1080125047 >> v2:172.21.15.114:3300/0 conn(0x7f6f5815f490 0x7f6f580ad010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.791+0000 7f6f5cd24640 1 -- 172.21.15.60:0/1080125047 >> v1:172.21.15.60:6789/0 conn(0x7f6f5806f0c0 legacy=0x7f6f580af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:36.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.791+0000 7f6f4ffff640 1 -- 172.21.15.60:0/1080125047 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3072099908 0 0) 0x7f6f580ad6f0 con 0x7f6f5806f0c0 2024-05-09T05:00:36.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.791+0000 7f6f5cd24640 1 -- 172.21.15.60:0/1080125047 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f58071af0 con 0x7f6f580737a0 2024-05-09T05:00:36.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.792+0000 7f6f5cd24640 1 --2- 172.21.15.60:0/1080125047 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f580737a0 0x7f6f580722f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6f580b2780 tx=0x7f6f4802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef64269b72d3b1e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:36.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.792+0000 7f6f4ffff640 1 -- 172.21.15.60:0/1080125047 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f48037070 con 0x7f6f580737a0 2024-05-09T05:00:36.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.792+0000 7f6f4ffff640 1 -- 172.21.15.60:0/1080125047 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f48032440 con 0x7f6f580737a0 2024-05-09T05:00:36.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.792+0000 7f6f4ffff640 1 -- 172.21.15.60:0/1080125047 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f48032810 con 0x7f6f580737a0 2024-05-09T05:00:36.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.793+0000 7f6f5f789640 1 -- 172.21.15.60:0/1080125047 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f580737a0 msgr2=0x7f6f580722f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:36.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.793+0000 7f6f5f789640 1 --2- 172.21.15.60:0/1080125047 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f580737a0 0x7f6f580722f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6f580b2780 tx=0x7f6f4802a1a0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.793+0000 7f6f5f789640 1 -- 172.21.15.60:0/1080125047 shutdown_connections 2024-05-09T05:00:36.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.793+0000 7f6f5f789640 1 --2- 172.21.15.60:0/1080125047 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f580737a0 0x7f6f580722f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.793+0000 7f6f5f789640 1 --2- 172.21.15.60:0/1080125047 >> v2:172.21.15.114:3300/0 conn(0x7f6f5815f490 0x7f6f580ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.793+0000 7f6f5f789640 1 -- 172.21.15.60:0/1080125047 >> 172.21.15.60:0/1080125047 conn(0x7f6f5806d7e0 msgr2=0x7f6f580b29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:36.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.793+0000 7f6f5f789640 1 -- 172.21.15.60:0/1080125047 shutdown_connections 2024-05-09T05:00:36.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.793+0000 7f6f5f789640 1 -- 172.21.15.60:0/1080125047 wait complete. 2024-05-09T05:00:36.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.794+0000 7f6f5f789640 1 Processor -- start 2024-05-09T05:00:36.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.794+0000 7f6f5f789640 1 -- start start 2024-05-09T05:00:36.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.795+0000 7f6f5f789640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f5806f0c0 0x7f6f5814c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:36.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.795+0000 7f6f5f789640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f580737a0 0x7f6f580c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:36.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.795+0000 7f6f5f789640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5815f490 0x7f6f580b1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:36.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.795+0000 7f6f5f789640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6f58072830 con 0x7f6f5815f490 2024-05-09T05:00:36.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.795+0000 7f6f5f789640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6f58073c00 con 0x7f6f580737a0 2024-05-09T05:00:36.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.795+0000 7f6f5f789640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6f5806f530 con 0x7f6f5806f0c0 2024-05-09T05:00:36.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.795+0000 7f6f5dd26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5815f490 0x7f6f580b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:36.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.795+0000 7f6f5dd26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5815f490 0x7f6f580b1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35166/0 (socket says 172.21.15.60:35166) 2024-05-09T05:00:36.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.795+0000 7f6f5dd26640 1 -- 172.21.15.60:0/3517899836 learned_addr learned my addr 172.21.15.60:0/3517899836 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:36.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.795+0000 7f6f5cd24640 1 --2- 172.21.15.60:0/3517899836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f580737a0 0x7f6f580c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:36.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.795+0000 7f6f5d525640 1 --2- 172.21.15.60:0/3517899836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f5806f0c0 0x7f6f5814c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:36.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.795+0000 7f6f5dd26640 1 -- 172.21.15.60:0/3517899836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f5806f0c0 msgr2=0x7f6f5814c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:36.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.795+0000 7f6f5dd26640 1 --2- 172.21.15.60:0/3517899836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f5806f0c0 0x7f6f5814c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.795+0000 7f6f5dd26640 1 -- 172.21.15.60:0/3517899836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f580737a0 msgr2=0x7f6f580c88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:36.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.795+0000 7f6f5dd26640 1 --2- 172.21.15.60:0/3517899836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f580737a0 0x7f6f580c88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:36.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.795+0000 7f6f5dd26640 1 -- 172.21.15.60:0/3517899836 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f581f62f0 con 0x7f6f5815f490 2024-05-09T05:00:36.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.795+0000 7f6f5dd26640 1 --2- 172.21.15.60:0/3517899836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5815f490 0x7f6f580b1200 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6f5000c690 tx=0x7f6f5000cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:36.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.795+0000 7f6f4dffb640 1 -- 172.21.15.60:0/3517899836 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f50018070 con 0x7f6f5815f490 2024-05-09T05:00:36.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.796+0000 7f6f4dffb640 1 -- 172.21.15.60:0/3517899836 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f50004d60 con 0x7f6f5815f490 2024-05-09T05:00:36.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.796+0000 7f6f5f789640 1 -- 172.21.15.60:0/3517899836 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f581f65d0 con 0x7f6f5815f490 2024-05-09T05:00:36.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.796+0000 7f6f4dffb640 1 -- 172.21.15.60:0/3517899836 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5000a880 con 0x7f6f5815f490 2024-05-09T05:00:36.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.796+0000 7f6f5f789640 1 -- 172.21.15.60:0/3517899836 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f581f6b90 con 0x7f6f5815f490 2024-05-09T05:00:36.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.797+0000 7f6f5f789640 1 -- 172.21.15.60:0/3517899836 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f58051040 con 0x7f6f5815f490 2024-05-09T05:00:36.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.797+0000 7f6f4dffb640 1 -- 172.21.15.60:0/3517899836 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6f5001b070 con 0x7f6f5815f490 2024-05-09T05:00:36.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.798+0000 7f6f4dffb640 1 --2- 172.21.15.60:0/3517899836 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6f3c0762d0 0x7f6f3c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:36.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.798+0000 7f6f4dffb640 1 -- 172.21.15.60:0/3517899836 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6f50096d10 con 0x7f6f5815f490 2024-05-09T05:00:36.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.798+0000 7f6f5d525640 1 --2- 172.21.15.60:0/3517899836 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6f3c0762d0 0x7f6f3c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:36.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.799+0000 7f6f5d525640 1 --2- 172.21.15.60:0/3517899836 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6f3c0762d0 0x7f6f3c078790 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6f58162360 tx=0x7f6f54008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:36.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:36.802+0000 7f6f4dffb640 1 -- 172.21.15.60:0/3517899836 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6f5005fd70 con 0x7f6f5815f490 2024-05-09T05:00:37.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.036+0000 7f6f5f789640 1 -- 172.21.15.60:0/3517899836 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f58070340 con 0x7f6f5815f490 2024-05-09T05:00:37.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.036+0000 7f6f4dffb640 1 -- 172.21.15.60:0/3517899836 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7f6f50064070 con 0x7f6f5815f490 2024-05-09T05:00:37.037 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:37.037 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1285340937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1285340937},{"type":"v1","addr":"172.21.15.60:6833","nonce":1285340937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2599193390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2599193390},{"type":"v1","addr":"172.21.15.114:6835","nonce":2599193390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3432609307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3432609307},{"type":"v1","addr":"172.21.15.114:6837","nonce":3432609307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:00:12.208556+0000","modified":"2024-05-09T05:00:31.347095+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2135423314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2135423314},{"type":"v1","addr":"172.21.15.60:6835","nonce":2135423314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-05-09T05:00:37.037 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2024-05-09T05:00:37.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.039+0000 7f6f5f789640 1 -- 172.21.15.60:0/3517899836 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6f3c0762d0 msgr2=0x7f6f3c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.039+0000 7f6f5f789640 1 --2- 172.21.15.60:0/3517899836 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6f3c0762d0 0x7f6f3c078790 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6f58162360 tx=0x7f6f54008040 comp rx=0 tx=0).stop 2024-05-09T05:00:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.039+0000 7f6f5f789640 1 -- 172.21.15.60:0/3517899836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5815f490 msgr2=0x7f6f580b1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.039+0000 7f6f5f789640 1 --2- 172.21.15.60:0/3517899836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5815f490 0x7f6f580b1200 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6f5000c690 tx=0x7f6f5000cb50 comp rx=0 tx=0).stop 2024-05-09T05:00:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.039+0000 7f6f5f789640 1 -- 172.21.15.60:0/3517899836 shutdown_connections 2024-05-09T05:00:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.039+0000 7f6f5f789640 1 --2- 172.21.15.60:0/3517899836 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6f3c0762d0 0x7f6f3c078790 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.039+0000 7f6f5f789640 1 --2- 172.21.15.60:0/3517899836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6f5815f490 0x7f6f580b1200 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.039+0000 7f6f5f789640 1 --2- 172.21.15.60:0/3517899836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6f580737a0 0x7f6f580c88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.039+0000 7f6f5f789640 1 --2- 172.21.15.60:0/3517899836 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6f5806f0c0 0x7f6f5814c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.039+0000 7f6f5f789640 1 -- 172.21.15.60:0/3517899836 >> 172.21.15.60:0/3517899836 conn(0x7f6f5806d7e0 msgr2=0x7f6f5815d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.039+0000 7f6f5f789640 1 -- 172.21.15.60:0/3517899836 shutdown_connections 2024-05-09T05:00:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.039+0000 7f6f5f789640 1 -- 172.21.15.60:0/3517899836 wait complete. 2024-05-09T05:00:37.050 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:00:37.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.154+0000 7f9a3d9d0640 1 Processor -- start 2024-05-09T05:00:37.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.154+0000 7f9a3d9d0640 1 -- start start 2024-05-09T05:00:37.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.155+0000 7f9a3d9d0640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a3806fe20 0x7f9a380702a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.155+0000 7f9a3d9d0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9a380736d0 0x7f9a38073b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.155+0000 7f9a3d9d0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a380d08e0 con 0x7f9a3815f300 2024-05-09T05:00:37.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.155+0000 7f9a3d9d0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f9a38073240 con 0x7f9a380736d0 2024-05-09T05:00:37.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.155+0000 7f9a3d9d0640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9a380d0bc0 con 0x7f9a3806fe20 2024-05-09T05:00:37.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.155+0000 7f9a377fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9a380736d0 0x7f9a38073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.155+0000 7f9a36ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9a3815f300 0x7f9a38072220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38546/0 (socket says 172.21.15.60:38546) 2024-05-09T05:00:37.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.155+0000 7f9a377fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9a380736d0 0x7f9a38073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35182/0 (socket says 172.21.15.60:35182) 2024-05-09T05:00:37.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.155+0000 7f9a36ffd640 1 -- 172.21.15.60:0/3025646147 learned_addr learned my addr 172.21.15.60:0/3025646147 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:37.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.155+0000 7f9a367fc640 1 --2- 172.21.15.60:0/3025646147 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a3806fe20 0x7f9a380702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.155+0000 7f9a377fe640 1 -- 172.21.15.60:0/3025646147 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a3806fe20 msgr2=0x7f9a380702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:37.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.155+0000 7f9a377fe640 1 --2- 172.21.15.60:0/3025646147 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a3806fe20 0x7f9a380702a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.155+0000 7f9a377fe640 1 -- 172.21.15.60:0/3025646147 >> v1:172.21.15.60:6789/0 conn(0x7f9a3815f300 legacy=0x7f9a38072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:37.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.155+0000 7f9a377fe640 1 -- 172.21.15.60:0/3025646147 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a38072ed0 con 0x7f9a380736d0 2024-05-09T05:00:37.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.155+0000 7f9a35ffb640 1 -- 172.21.15.60:0/3025646147 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2237321754 0 0) 0x7f9a380d08e0 con 0x7f9a3815f300 2024-05-09T05:00:37.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.156+0000 7f9a377fe640 1 --2- 172.21.15.60:0/3025646147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a380736d0 0x7f9a38073b30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9a30009d90 tx=0x7f9a3002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e122b505da6855 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:37.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.156+0000 7f9a35ffb640 1 -- 172.21.15.60:0/3025646147 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a30007740 con 0x7f9a380736d0 2024-05-09T05:00:37.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.156+0000 7f9a35ffb640 1 -- 172.21.15.60:0/3025646147 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a300078e0 con 0x7f9a380736d0 2024-05-09T05:00:37.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.156+0000 7f9a35ffb640 1 -- 172.21.15.60:0/3025646147 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a30007cb0 con 0x7f9a380736d0 2024-05-09T05:00:37.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.157+0000 7f9a3d9d0640 1 -- 172.21.15.60:0/3025646147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a380736d0 msgr2=0x7f9a38073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:37.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.157+0000 7f9a3d9d0640 1 --2- 172.21.15.60:0/3025646147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a380736d0 0x7f9a38073b30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9a30009d90 tx=0x7f9a3002a660 comp rx=0 tx=0).stop 2024-05-09T05:00:37.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.157+0000 7f9a3d9d0640 1 -- 172.21.15.60:0/3025646147 shutdown_connections 2024-05-09T05:00:37.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.157+0000 7f9a3d9d0640 1 --2- 172.21.15.60:0/3025646147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a380736d0 0x7f9a38073b30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.157+0000 7f9a3d9d0640 1 --2- 172.21.15.60:0/3025646147 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a3806fe20 0x7f9a380702a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.157+0000 7f9a3d9d0640 1 -- 172.21.15.60:0/3025646147 >> 172.21.15.60:0/3025646147 conn(0x7f9a380b4ef0 msgr2=0x7f9a3806e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:37.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.157+0000 7f9a3d9d0640 1 -- 172.21.15.60:0/3025646147 shutdown_connections 2024-05-09T05:00:37.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.157+0000 7f9a3d9d0640 1 -- 172.21.15.60:0/3025646147 wait complete. 2024-05-09T05:00:37.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.158+0000 7f9a3d9d0640 1 Processor -- start 2024-05-09T05:00:37.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.158+0000 7f9a3d9d0640 1 -- start start 2024-05-09T05:00:37.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.159+0000 7f9a3d9d0640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a3806fe20 0x7f9a381f7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.159+0000 7f9a3d9d0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a380736d0 0x7f9a380d3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.159+0000 7f9a3d9d0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a3815f300 0x7f9a380d4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.159+0000 7f9a3d9d0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9a380d04f0 con 0x7f9a380736d0 2024-05-09T05:00:37.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.159+0000 7f9a3d9d0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a3809c1e0 con 0x7f9a3815f300 2024-05-09T05:00:37.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.159+0000 7f9a3d9d0640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9a380d0670 con 0x7f9a3806fe20 2024-05-09T05:00:37.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.159+0000 7f9a367fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a380736d0 0x7f9a380d3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.159+0000 7f9a367fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a380736d0 0x7f9a380d3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35186/0 (socket says 172.21.15.60:35186) 2024-05-09T05:00:37.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.159+0000 7f9a367fc640 1 -- 172.21.15.60:0/469147733 learned_addr learned my addr 172.21.15.60:0/469147733 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:37.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.159+0000 7f9a377fe640 1 --2- 172.21.15.60:0/469147733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a3815f300 0x7f9a380d4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.159+0000 7f9a367fc640 1 -- 172.21.15.60:0/469147733 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a3806fe20 msgr2=0x7f9a381f7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:37.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.159+0000 7f9a367fc640 1 --2- 172.21.15.60:0/469147733 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a3806fe20 0x7f9a381f7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.159+0000 7f9a367fc640 1 -- 172.21.15.60:0/469147733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a3815f300 msgr2=0x7f9a380d4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:37.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.159+0000 7f9a367fc640 1 --2- 172.21.15.60:0/469147733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a3815f300 0x7f9a380d4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.159+0000 7f9a367fc640 1 -- 172.21.15.60:0/469147733 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a381f83d0 con 0x7f9a380736d0 2024-05-09T05:00:37.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.159+0000 7f9a367fc640 1 --2- 172.21.15.60:0/469147733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a380736d0 0x7f9a380d3b00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c00d800 tx=0x7f9a2c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:37.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.160+0000 7f9a1ffff640 1 -- 172.21.15.60:0/469147733 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c00b6c0 con 0x7f9a380736d0 2024-05-09T05:00:37.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.160+0000 7f9a1ffff640 1 -- 172.21.15.60:0/469147733 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a2c019070 con 0x7f9a380736d0 2024-05-09T05:00:37.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.160+0000 7f9a1ffff640 1 -- 172.21.15.60:0/469147733 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c00ba80 con 0x7f9a380736d0 2024-05-09T05:00:37.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.160+0000 7f9a3d9d0640 1 -- 172.21.15.60:0/469147733 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a38161770 con 0x7f9a380736d0 2024-05-09T05:00:37.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.160+0000 7f9a3d9d0640 1 -- 172.21.15.60:0/469147733 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a38161d30 con 0x7f9a380736d0 2024-05-09T05:00:37.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.161+0000 7f9a3d9d0640 1 -- 172.21.15.60:0/469147733 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a38073e90 con 0x7f9a380736d0 2024-05-09T05:00:37.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.161+0000 7f9a1ffff640 1 -- 172.21.15.60:0/469147733 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9a2c00bc20 con 0x7f9a380736d0 2024-05-09T05:00:37.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.162+0000 7f9a1ffff640 1 --2- 172.21.15.60:0/469147733 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9a240762d0 0x7f9a24078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.162+0000 7f9a1ffff640 1 -- 172.21.15.60:0/469147733 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9a2c097db0 con 0x7f9a380736d0 2024-05-09T05:00:37.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.162+0000 7f9a36ffd640 1 --2- 172.21.15.60:0/469147733 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9a240762d0 0x7f9a24078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.163+0000 7f9a36ffd640 1 --2- 172.21.15.60:0/469147733 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9a240762d0 0x7f9a24078790 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9a28007fd0 tx=0x7f9a28007960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:37.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.166+0000 7f9a1ffff640 1 -- 172.21.15.60:0/469147733 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9a38073e90 con 0x7f9a380736d0 2024-05-09T05:00:37.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.396+0000 7f9a3d9d0640 1 -- 172.21.15.60:0/469147733 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a38051040 con 0x7f9a380736d0 2024-05-09T05:00:37.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.396+0000 7f9a1ffff640 1 -- 172.21.15.60:0/469147733 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7f9a2c061460 con 0x7f9a380736d0 2024-05-09T05:00:37.397 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:37.397 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1285340937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1285340937},{"type":"v1","addr":"172.21.15.60:6833","nonce":1285340937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2599193390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2599193390},{"type":"v1","addr":"172.21.15.114:6835","nonce":2599193390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3432609307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3432609307},{"type":"v1","addr":"172.21.15.114:6837","nonce":3432609307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:00:12.208556+0000","modified":"2024-05-09T05:00:31.347095+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2135423314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2135423314},{"type":"v1","addr":"172.21.15.60:6835","nonce":2135423314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-05-09T05:00:37.398 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2024-05-09T05:00:37.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.398+0000 7f9a3d9d0640 1 -- 172.21.15.60:0/469147733 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9a240762d0 msgr2=0x7f9a24078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:37.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.398+0000 7f9a3d9d0640 1 --2- 172.21.15.60:0/469147733 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9a240762d0 0x7f9a24078790 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9a28007fd0 tx=0x7f9a28007960 comp rx=0 tx=0).stop 2024-05-09T05:00:37.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.398+0000 7f9a3d9d0640 1 -- 172.21.15.60:0/469147733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a380736d0 msgr2=0x7f9a380d3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:37.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.398+0000 7f9a3d9d0640 1 --2- 172.21.15.60:0/469147733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a380736d0 0x7f9a380d3b00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c00d800 tx=0x7f9a2c00dcc0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.399+0000 7f9a3d9d0640 1 -- 172.21.15.60:0/469147733 shutdown_connections 2024-05-09T05:00:37.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.399+0000 7f9a3d9d0640 1 --2- 172.21.15.60:0/469147733 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9a240762d0 0x7f9a24078790 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.399+0000 7f9a3d9d0640 1 --2- 172.21.15.60:0/469147733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a3815f300 0x7f9a380d4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.399+0000 7f9a3d9d0640 1 --2- 172.21.15.60:0/469147733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a380736d0 0x7f9a380d3b00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.399+0000 7f9a3d9d0640 1 --2- 172.21.15.60:0/469147733 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9a3806fe20 0x7f9a381f7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.399+0000 7f9a3d9d0640 1 -- 172.21.15.60:0/469147733 >> 172.21.15.60:0/469147733 conn(0x7f9a380b4ef0 msgr2=0x7f9a3806e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:37.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.399+0000 7f9a3d9d0640 1 -- 172.21.15.60:0/469147733 shutdown_connections 2024-05-09T05:00:37.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.399+0000 7f9a3d9d0640 1 -- 172.21.15.60:0/469147733 wait complete. 2024-05-09T05:00:37.411 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:00:37.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.514+0000 7f7d75d72640 1 Processor -- start 2024-05-09T05:00:37.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.514+0000 7f7d75d72640 1 -- start start 2024-05-09T05:00:37.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.515+0000 7f7d75d72640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7d7015f300 0x7f7d701639f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.515+0000 7f7d75d72640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d7006f9f0 0x7f7d700d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.515+0000 7f7d75d72640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7d700d3930 0x7f7d7006ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.515+0000 7f7d75d72640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f7d7006ff60 con 0x7f7d7015f300 2024-05-09T05:00:37.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.515+0000 7f7d75d72640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f7d70070240 con 0x7f7d700d3930 2024-05-09T05:00:37.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.515+0000 7f7d75d72640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7d70070520 con 0x7f7d7006f9f0 2024-05-09T05:00:37.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.515+0000 7f7d6f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7d7015f300 0x7f7d701639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.515+0000 7f7d6f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7d7015f300 0x7f7d701639f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35198/0 (socket says 172.21.15.60:35198) 2024-05-09T05:00:37.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.515+0000 7f7d6f7fe640 1 -- 172.21.15.60:0/2213146366 learned_addr learned my addr 172.21.15.60:0/2213146366 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:37.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.515+0000 7f7d6ffff640 1 --2- 172.21.15.60:0/2213146366 >> v2:172.21.15.114:3300/0 conn(0x7f7d700d3930 0x7f7d7006ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.515+0000 7f7d6f7fe640 1 -- 172.21.15.60:0/2213146366 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d7006f9f0 msgr2=0x7f7d700d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:37.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.515+0000 7f7d6effd640 1 --2- 172.21.15.60:0/2213146366 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d7006f9f0 0x7f7d700d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.516+0000 7f7d6f7fe640 1 --2- 172.21.15.60:0/2213146366 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d7006f9f0 0x7f7d700d33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.516+0000 7f7d6f7fe640 1 -- 172.21.15.60:0/2213146366 >> v2:172.21.15.114:3300/0 conn(0x7f7d700d3930 msgr2=0x7f7d7006ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:37.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.516+0000 7f7d6f7fe640 1 --2- 172.21.15.60:0/2213146366 >> v2:172.21.15.114:3300/0 conn(0x7f7d700d3930 0x7f7d7006ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.516+0000 7f7d6f7fe640 1 -- 172.21.15.60:0/2213146366 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d7006e740 con 0x7f7d7015f300 2024-05-09T05:00:37.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.516+0000 7f7d6ffff640 1 --2- 172.21.15.60:0/2213146366 >> v2:172.21.15.114:3300/0 conn(0x7f7d700d3930 0x7f7d7006ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:37.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.516+0000 7f7d6effd640 1 --2- 172.21.15.60:0/2213146366 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d7006f9f0 0x7f7d700d33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:37.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.516+0000 7f7d6f7fe640 1 --2- 172.21.15.60:0/2213146366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d7015f300 0x7f7d701639f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7d7009f890 tx=0x7f7d6002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc46e785f52682d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:37.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.516+0000 7f7d6e7fc640 1 -- 172.21.15.60:0/2213146366 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d60037070 con 0x7f7d7015f300 2024-05-09T05:00:37.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.516+0000 7f7d6e7fc640 1 -- 172.21.15.60:0/2213146366 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d60032440 con 0x7f7d7015f300 2024-05-09T05:00:37.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.516+0000 7f7d6e7fc640 1 -- 172.21.15.60:0/2213146366 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d60032810 con 0x7f7d7015f300 2024-05-09T05:00:37.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.517+0000 7f7d75d72640 1 -- 172.21.15.60:0/2213146366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d7015f300 msgr2=0x7f7d701639f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:37.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.517+0000 7f7d75d72640 1 --2- 172.21.15.60:0/2213146366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d7015f300 0x7f7d701639f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7d7009f890 tx=0x7f7d6002a1e0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.517+0000 7f7d75d72640 1 -- 172.21.15.60:0/2213146366 shutdown_connections 2024-05-09T05:00:37.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.517+0000 7f7d75d72640 1 --2- 172.21.15.60:0/2213146366 >> v2:172.21.15.114:3300/0 conn(0x7f7d700d3930 0x7f7d7006ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.517+0000 7f7d75d72640 1 --2- 172.21.15.60:0/2213146366 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d7006f9f0 0x7f7d700d33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.517+0000 7f7d75d72640 1 --2- 172.21.15.60:0/2213146366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d7015f300 0x7f7d701639f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.517+0000 7f7d75d72640 1 -- 172.21.15.60:0/2213146366 >> 172.21.15.60:0/2213146366 conn(0x7f7d7009b850 msgr2=0x7f7d7009bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:37.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.517+0000 7f7d75d72640 1 -- 172.21.15.60:0/2213146366 shutdown_connections 2024-05-09T05:00:37.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.517+0000 7f7d75d72640 1 -- 172.21.15.60:0/2213146366 wait complete. 2024-05-09T05:00:37.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.518+0000 7f7d75d72640 1 Processor -- start 2024-05-09T05:00:37.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.518+0000 7f7d75d72640 1 -- start start 2024-05-09T05:00:37.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.519+0000 7f7d75d72640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d7006f9f0 0x7f7d701ea0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.519+0000 7f7d75d72640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d700d3930 0x7f7d701eb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.519+0000 7f7d75d72640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d7015f300 0x7f7d7009a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.519+0000 7f7d75d72640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7006f480 con 0x7f7d7006f9f0 2024-05-09T05:00:37.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.519+0000 7f7d75d72640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7d70163f30 con 0x7f7d7015f300 2024-05-09T05:00:37.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.519+0000 7f7d75d72640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7d7006f600 con 0x7f7d700d3930 2024-05-09T05:00:37.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.519+0000 7f7d6f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d7006f9f0 0x7f7d701ea0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.519+0000 7f7d6f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d7006f9f0 0x7f7d701ea0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35206/0 (socket says 172.21.15.60:35206) 2024-05-09T05:00:37.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.519+0000 7f7d6f7fe640 1 -- 172.21.15.60:0/934125938 learned_addr learned my addr 172.21.15.60:0/934125938 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:37.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.519+0000 7f7d6f7fe640 1 -- 172.21.15.60:0/934125938 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d700d3930 msgr2=0x7f7d701eb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:37.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.519+0000 7f7d6f7fe640 1 --2- 172.21.15.60:0/934125938 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d700d3930 0x7f7d701eb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.519+0000 7f7d6f7fe640 1 -- 172.21.15.60:0/934125938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d7015f300 msgr2=0x7f7d7009a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:37.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.519+0000 7f7d6ffff640 1 --2- 172.21.15.60:0/934125938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d7015f300 0x7f7d7009a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.519+0000 7f7d6f7fe640 1 --2- 172.21.15.60:0/934125938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d7015f300 0x7f7d7009a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.520+0000 7f7d6f7fe640 1 -- 172.21.15.60:0/934125938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d701ea5f0 con 0x7f7d7006f9f0 2024-05-09T05:00:37.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.520+0000 7f7d6ffff640 1 --2- 172.21.15.60:0/934125938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d7015f300 0x7f7d7009a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:37.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.520+0000 7f7d6f7fe640 1 --2- 172.21.15.60:0/934125938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d7006f9f0 0x7f7d701ea0b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7d6002a220 tx=0x7f7d600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:37.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.520+0000 7f7d57fff640 1 -- 172.21.15.60:0/934125938 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d60037040 con 0x7f7d7006f9f0 2024-05-09T05:00:37.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.520+0000 7f7d57fff640 1 -- 172.21.15.60:0/934125938 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d60042070 con 0x7f7d7006f9f0 2024-05-09T05:00:37.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.520+0000 7f7d75d72640 1 -- 172.21.15.60:0/934125938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d701e9010 con 0x7f7d7006f9f0 2024-05-09T05:00:37.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.520+0000 7f7d57fff640 1 -- 172.21.15.60:0/934125938 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d60030bb0 con 0x7f7d7006f9f0 2024-05-09T05:00:37.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.520+0000 7f7d75d72640 1 -- 172.21.15.60:0/934125938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d701e95d0 con 0x7f7d7006f9f0 2024-05-09T05:00:37.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.521+0000 7f7d75d72640 1 -- 172.21.15.60:0/934125938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d70051040 con 0x7f7d7006f9f0 2024-05-09T05:00:37.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.521+0000 7f7d57fff640 1 -- 172.21.15.60:0/934125938 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7d60047460 con 0x7f7d7006f9f0 2024-05-09T05:00:37.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.522+0000 7f7d57fff640 1 --2- 172.21.15.60:0/934125938 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7d5c0762d0 0x7f7d5c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.522+0000 7f7d57fff640 1 -- 172.21.15.60:0/934125938 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7d600b8590 con 0x7f7d7006f9f0 2024-05-09T05:00:37.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.522+0000 7f7d6effd640 1 --2- 172.21.15.60:0/934125938 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7d5c0762d0 0x7f7d5c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.522+0000 7f7d6effd640 1 --2- 172.21.15.60:0/934125938 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7d5c0762d0 0x7f7d5c078790 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7d64006fd0 tx=0x7f7d64008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:37.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.526+0000 7f7d57fff640 1 -- 172.21.15.60:0/934125938 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7d60081bb0 con 0x7f7d7006f9f0 2024-05-09T05:00:37.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.717+0000 7f7d75d72640 1 -- 172.21.15.60:0/934125938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7d700712c0 con 0x7f7d7006f9f0 2024-05-09T05:00:37.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.718+0000 7f7d57fff640 1 -- 172.21.15.60:0/934125938 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18490 (secure 0 0 0) 0x7f7d600814b0 con 0x7f7d7006f9f0 2024-05-09T05:00:37.719 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:37.719 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:00:17.248958+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:00:10.358133+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:00:11.157769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T05:00:14.850459+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:00:37.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.720+0000 7f7d75d72640 1 -- 172.21.15.60:0/934125938 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7d5c0762d0 msgr2=0x7f7d5c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:37.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.720+0000 7f7d75d72640 1 --2- 172.21.15.60:0/934125938 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7d5c0762d0 0x7f7d5c078790 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7d64006fd0 tx=0x7f7d64008040 comp rx=0 tx=0).stop 2024-05-09T05:00:37.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.720+0000 7f7d75d72640 1 -- 172.21.15.60:0/934125938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d7006f9f0 msgr2=0x7f7d701ea0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:37.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.720+0000 7f7d75d72640 1 --2- 172.21.15.60:0/934125938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d7006f9f0 0x7f7d701ea0b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7d6002a220 tx=0x7f7d600046c0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.720+0000 7f7d75d72640 1 -- 172.21.15.60:0/934125938 shutdown_connections 2024-05-09T05:00:37.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.720+0000 7f7d75d72640 1 --2- 172.21.15.60:0/934125938 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7d5c0762d0 0x7f7d5c078790 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.720+0000 7f7d75d72640 1 --2- 172.21.15.60:0/934125938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d7015f300 0x7f7d7009a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.720+0000 7f7d75d72640 1 --2- 172.21.15.60:0/934125938 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7d700d3930 0x7f7d701eb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.720+0000 7f7d75d72640 1 --2- 172.21.15.60:0/934125938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d7006f9f0 0x7f7d701ea0b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.720+0000 7f7d75d72640 1 -- 172.21.15.60:0/934125938 >> 172.21.15.60:0/934125938 conn(0x7f7d7009b850 msgr2=0x7f7d70162a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:37.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.721+0000 7f7d75d72640 1 -- 172.21.15.60:0/934125938 shutdown_connections 2024-05-09T05:00:37.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.721+0000 7f7d75d72640 1 -- 172.21.15.60:0/934125938 wait complete. 2024-05-09T05:00:37.732 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:00:37.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.836+0000 7fbf87209640 1 Processor -- start 2024-05-09T05:00:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.836+0000 7fbf87209640 1 -- start start 2024-05-09T05:00:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.837+0000 7fbf87209640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf800b2ea0 0x7fbf80098d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.837+0000 7fbf87209640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbf8009b640 0x7fbf8009bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.837+0000 7fbf87209640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf8015fad0 con 0x7fbf800b3280 2024-05-09T05:00:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.837+0000 7fbf87209640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fbf8009a880 con 0x7fbf8009b640 2024-05-09T05:00:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.837+0000 7fbf87209640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbf8009ab60 con 0x7fbf800b2ea0 2024-05-09T05:00:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.837+0000 7fbf7ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbf800b3280 0x7fbf800ae910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38562/0 (socket says 172.21.15.60:38562) 2024-05-09T05:00:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.837+0000 7fbf7ffff640 1 -- 172.21.15.60:0/3353767489 learned_addr learned my addr 172.21.15.60:0/3353767489 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf857a6640 1 --2- 172.21.15.60:0/3353767489 >> v2:172.21.15.114:3300/0 conn(0x7fbf8009b640 0x7fbf8009bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf84fa5640 1 --2- 172.21.15.60:0/3353767489 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf800b2ea0 0x7fbf80098d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf7f7fe640 1 -- 172.21.15.60:0/3353767489 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3154345015 0 0) 0x7fbf8015fad0 con 0x7fbf800b3280 2024-05-09T05:00:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf7f7fe640 1 -- 172.21.15.60:0/3353767489 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf68003520 con 0x7fbf800b3280 2024-05-09T05:00:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf7f7fe640 1 -- 172.21.15.60:0/3353767489 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 601569698 0 0) 0x7fbf68003520 con 0x7fbf800b3280 2024-05-09T05:00:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf7f7fe640 1 -- 172.21.15.60:0/3353767489 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf800b2ea0 msgr2=0x7fbf80098d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:37.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf7f7fe640 1 --2- 172.21.15.60:0/3353767489 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf800b2ea0 0x7fbf80098d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf7f7fe640 1 -- 172.21.15.60:0/3353767489 >> v2:172.21.15.114:3300/0 conn(0x7fbf8009b640 msgr2=0x7fbf8009bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:37.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf7f7fe640 1 --2- 172.21.15.60:0/3353767489 >> v2:172.21.15.114:3300/0 conn(0x7fbf8009b640 0x7fbf8009bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf7f7fe640 1 -- 172.21.15.60:0/3353767489 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8009ae40 con 0x7fbf800b3280 2024-05-09T05:00:37.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf7f7fe640 1 -- 172.21.15.60:0/3353767489 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7fbf740016f0 con 0x7fbf800b3280 2024-05-09T05:00:37.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf7f7fe640 1 -- 172.21.15.60:0/3353767489 >> v1:172.21.15.60:6789/0 conn(0x7fbf800b3280 legacy=0x7fbf800ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:37.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf7f7fe640 1 --2- 172.21.15.60:0/3353767489 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf800b2ea0 0x7fbf68003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf7f7fe640 1 --2- 172.21.15.60:0/3353767489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf68003fd0 0x7fbf68004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf7f7fe640 1 --2- 172.21.15.60:0/3353767489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf68004d40 0x7fbf680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf7f7fe640 1 -- 172.21.15.60:0/3353767489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8009a880 con 0x7fbf68004d40 2024-05-09T05:00:37.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf7f7fe640 1 -- 172.21.15.60:0/3353767489 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8009ab60 con 0x7fbf68003fd0 2024-05-09T05:00:37.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf7f7fe640 1 -- 172.21.15.60:0/3353767489 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbf6800baa0 con 0x7fbf800b2ea0 2024-05-09T05:00:37.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf857a6640 1 --2- 172.21.15.60:0/3353767489 >> v2:172.21.15.114:3300/0 conn(0x7fbf8009b640 0x7fbf8009bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:00:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf84fa5640 1 --2- 172.21.15.60:0/3353767489 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf800b2ea0 0x7fbf68003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf7ffff640 1 --2- 172.21.15.60:0/3353767489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf68003fd0 0x7fbf68004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.838+0000 7fbf857a6640 1 --2- 172.21.15.60:0/3353767489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf68004d40 0x7fbf680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.839+0000 7fbf857a6640 1 -- 172.21.15.60:0/3353767489 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf800b2ea0 msgr2=0x7fbf68003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.839+0000 7fbf857a6640 1 --2- 172.21.15.60:0/3353767489 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf800b2ea0 0x7fbf68003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.839+0000 7fbf857a6640 1 -- 172.21.15.60:0/3353767489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf68003fd0 msgr2=0x7fbf68004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.839+0000 7fbf857a6640 1 --2- 172.21.15.60:0/3353767489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf68003fd0 0x7fbf68004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.839+0000 7fbf857a6640 1 -- 172.21.15.60:0/3353767489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf68003520 con 0x7fbf68004d40 2024-05-09T05:00:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.839+0000 7fbf84fa5640 1 --2- 172.21.15.60:0/3353767489 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf800b2ea0 0x7fbf68003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:00:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.839+0000 7fbf857a6640 1 --2- 172.21.15.60:0/3353767489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf68004d40 0x7fbf680051c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbf70009c70 tx=0x7fbf68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c884b6225f93aba5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.839+0000 7fbf7f7fe640 1 -- 172.21.15.60:0/3353767489 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf70009840 con 0x7fbf68004d40 2024-05-09T05:00:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.839+0000 7fbf7f7fe640 1 -- 172.21.15.60:0/3353767489 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf70004020 con 0x7fbf68004d40 2024-05-09T05:00:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.839+0000 7fbf7f7fe640 1 -- 172.21.15.60:0/3353767489 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf700043d0 con 0x7fbf68004d40 2024-05-09T05:00:37.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.840+0000 7fbf87209640 1 -- 172.21.15.60:0/3353767489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf68004d40 msgr2=0x7fbf680051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:37.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.840+0000 7fbf87209640 1 --2- 172.21.15.60:0/3353767489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf68004d40 0x7fbf680051c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbf70009c70 tx=0x7fbf68005830 comp rx=0 tx=0).stop 2024-05-09T05:00:37.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.840+0000 7fbf857a6640 1 -- 172.21.15.60:0/3353767489 reap_dead start 2024-05-09T05:00:37.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.840+0000 7fbf87209640 1 -- 172.21.15.60:0/3353767489 shutdown_connections 2024-05-09T05:00:37.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.840+0000 7fbf87209640 1 -- 172.21.15.60:0/3353767489 >> 172.21.15.60:0/3353767489 conn(0x7fbf80071e80 msgr2=0x7fbf80072290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:37.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.840+0000 7fbf87209640 1 -- 172.21.15.60:0/3353767489 shutdown_connections 2024-05-09T05:00:37.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.840+0000 7fbf87209640 1 -- 172.21.15.60:0/3353767489 wait complete. 2024-05-09T05:00:37.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.841+0000 7fbf87209640 1 Processor -- start 2024-05-09T05:00:37.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.841+0000 7fbf87209640 1 -- start start 2024-05-09T05:00:37.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.842+0000 7fbf87209640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf68004d40 0x7fbf800d1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.842+0000 7fbf87209640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf80159d80 0x7fbf8015a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.842+0000 7fbf87209640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf8015ae60 0x7fbf801f5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.842+0000 7fbf87209640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbf80098d60 con 0x7fbf80159d80 2024-05-09T05:00:37.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.842+0000 7fbf87209640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbf800b3700 con 0x7fbf68004d40 2024-05-09T05:00:37.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.842+0000 7fbf87209640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbf80098ee0 con 0x7fbf8015ae60 2024-05-09T05:00:37.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.842+0000 7fbf7ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf80159d80 0x7fbf8015a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.842+0000 7fbf7ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf80159d80 0x7fbf8015a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35230/0 (socket says 172.21.15.60:35230) 2024-05-09T05:00:37.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.842+0000 7fbf7ffff640 1 -- 172.21.15.60:0/3530615108 learned_addr learned my addr 172.21.15.60:0/3530615108 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:37.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.843+0000 7fbf84fa5640 1 --2- 172.21.15.60:0/3530615108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf68004d40 0x7fbf800d1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.843+0000 7fbf857a6640 1 --2- 172.21.15.60:0/3530615108 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf8015ae60 0x7fbf801f5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.843+0000 7fbf7ffff640 1 -- 172.21.15.60:0/3530615108 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf8015ae60 msgr2=0x7fbf801f5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:37.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.843+0000 7fbf7ffff640 1 --2- 172.21.15.60:0/3530615108 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf8015ae60 0x7fbf801f5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.843+0000 7fbf7ffff640 1 -- 172.21.15.60:0/3530615108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf68004d40 msgr2=0x7fbf800d1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:37.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.843+0000 7fbf7ffff640 1 --2- 172.21.15.60:0/3530615108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf68004d40 0x7fbf800d1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:37.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.843+0000 7fbf7ffff640 1 -- 172.21.15.60:0/3530615108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf80158fc0 con 0x7fbf80159d80 2024-05-09T05:00:37.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.843+0000 7fbf84fa5640 1 --2- 172.21.15.60:0/3530615108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf68004d40 0x7fbf800d1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:37.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.843+0000 7fbf857a6640 1 --2- 172.21.15.60:0/3530615108 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf8015ae60 0x7fbf801f5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:37.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.843+0000 7fbf7ffff640 1 --2- 172.21.15.60:0/3530615108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf80159d80 0x7fbf8015a920 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbf740072c0 tx=0x7fbf74007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:37.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.843+0000 7fbf7d7fa640 1 -- 172.21.15.60:0/3530615108 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7400f9c0 con 0x7fbf80159d80 2024-05-09T05:00:37.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.843+0000 7fbf7d7fa640 1 -- 172.21.15.60:0/3530615108 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf74018070 con 0x7fbf80159d80 2024-05-09T05:00:37.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.843+0000 7fbf87209640 1 -- 172.21.15.60:0/3530615108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf801592a0 con 0x7fbf80159d80 2024-05-09T05:00:37.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.843+0000 7fbf7d7fa640 1 -- 172.21.15.60:0/3530615108 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7400fda0 con 0x7fbf80159d80 2024-05-09T05:00:37.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.843+0000 7fbf87209640 1 -- 172.21.15.60:0/3530615108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf8015b310 con 0x7fbf80159d80 2024-05-09T05:00:37.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.844+0000 7fbf87209640 1 -- 172.21.15.60:0/3530615108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf80159690 con 0x7fbf80159d80 2024-05-09T05:00:37.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.845+0000 7fbf7d7fa640 1 -- 172.21.15.60:0/3530615108 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbf7401d460 con 0x7fbf80159d80 2024-05-09T05:00:37.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.845+0000 7fbf7d7fa640 1 --2- 172.21.15.60:0/3530615108 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbf6c0762d0 0x7fbf6c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:37.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.845+0000 7fbf7d7fa640 1 -- 172.21.15.60:0/3530615108 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fbf740979e0 con 0x7fbf80159d80 2024-05-09T05:00:37.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.845+0000 7fbf84fa5640 1 --2- 172.21.15.60:0/3530615108 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbf6c0762d0 0x7fbf6c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:37.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.846+0000 7fbf84fa5640 1 --2- 172.21.15.60:0/3530615108 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbf6c0762d0 0x7fbf6c078790 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbf780075e0 tx=0x7fbf7800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:37.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:37.849+0000 7fbf7d7fa640 1 -- 172.21.15.60:0/3530615108 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbf74061000 con 0x7fbf80159d80 2024-05-09T05:00:38.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.080+0000 7fbf87209640 1 -- 172.21.15.60:0/3530615108 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf80159470 con 0x7fbf80159d80 2024-05-09T05:00:38.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.081+0000 7fbf7d7fa640 1 -- 172.21.15.60:0/3530615108 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7fbf74060900 con 0x7fbf80159d80 2024-05-09T05:00:38.082 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:38.082 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1285340937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1285340937},{"type":"v1","addr":"172.21.15.60:6833","nonce":1285340937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2599193390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2599193390},{"type":"v1","addr":"172.21.15.114:6835","nonce":2599193390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3432609307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3432609307},{"type":"v1","addr":"172.21.15.114:6837","nonce":3432609307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:00:12.208556+0000","modified":"2024-05-09T05:00:31.347095+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2135423314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2135423314},{"type":"v1","addr":"172.21.15.60:6835","nonce":2135423314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-05-09T05:00:38.082 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2024-05-09T05:00:38.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.083+0000 7fbf87209640 1 -- 172.21.15.60:0/3530615108 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbf6c0762d0 msgr2=0x7fbf6c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:38.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.083+0000 7fbf87209640 1 --2- 172.21.15.60:0/3530615108 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbf6c0762d0 0x7fbf6c078790 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbf780075e0 tx=0x7fbf7800b040 comp rx=0 tx=0).stop 2024-05-09T05:00:38.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.083+0000 7fbf87209640 1 -- 172.21.15.60:0/3530615108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf80159d80 msgr2=0x7fbf8015a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:38.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.083+0000 7fbf87209640 1 --2- 172.21.15.60:0/3530615108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf80159d80 0x7fbf8015a920 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbf740072c0 tx=0x7fbf74007780 comp rx=0 tx=0).stop 2024-05-09T05:00:38.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.083+0000 7fbf87209640 1 -- 172.21.15.60:0/3530615108 shutdown_connections 2024-05-09T05:00:38.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.083+0000 7fbf87209640 1 --2- 172.21.15.60:0/3530615108 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbf6c0762d0 0x7fbf6c078790 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.083+0000 7fbf87209640 1 --2- 172.21.15.60:0/3530615108 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbf8015ae60 0x7fbf801f5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.083+0000 7fbf87209640 1 --2- 172.21.15.60:0/3530615108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf80159d80 0x7fbf8015a920 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.083+0000 7fbf87209640 1 --2- 172.21.15.60:0/3530615108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf68004d40 0x7fbf800d1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.083+0000 7fbf87209640 1 -- 172.21.15.60:0/3530615108 >> 172.21.15.60:0/3530615108 conn(0x7fbf80071e80 msgr2=0x7fbf8015f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:38.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.083+0000 7fbf87209640 1 -- 172.21.15.60:0/3530615108 shutdown_connections 2024-05-09T05:00:38.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.083+0000 7fbf87209640 1 -- 172.21.15.60:0/3530615108 wait complete. 2024-05-09T05:00:38.497 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T05:00:12.208556+0000', 'modified': '2024-05-09T05:00:31.347095+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4261}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4261': {'gid': 4261, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6835/2135423314', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6834', 'nonce': 2135423314}, {'type': 'v1', 'addr': '172.21.15.60:6835', 'nonce': 2135423314}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4261, 'qdb_cluster': [4261]} 2024-05-09T05:00:38.498 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T05:00:38.498 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T05:00:38.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.608+0000 7f663c946640 1 Processor -- start 2024-05-09T05:00:38.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.608+0000 7f663c946640 1 -- start start 2024-05-09T05:00:38.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.609+0000 7f663c946640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f663815c3a0 0x7f6638070150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.609+0000 7f663c946640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6638071b80 con 0x7f66380ac430 2024-05-09T05:00:38.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.609+0000 7f663c946640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6638071e60 con 0x7f6638073c30 2024-05-09T05:00:38.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.609+0000 7f663c946640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f6638072140 con 0x7f663815c3a0 2024-05-09T05:00:38.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.609+0000 7f6635d9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f66380ac430 0x7f6638099a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38572/0 (socket says 172.21.15.60:38572) 2024-05-09T05:00:38.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.609+0000 7f6635d9b640 1 -- 172.21.15.60:0/2765757555 learned_addr learned my addr 172.21.15.60:0/2765757555 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:38.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.609+0000 7f663659c640 1 --2- 172.21.15.60:0/2765757555 >> v2:172.21.15.114:3300/0 conn(0x7f663815c3a0 0x7f6638070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.609+0000 7f663559a640 1 -- 172.21.15.60:0/2765757555 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 238501336 0 0) 0x7f6638071b80 con 0x7f66380ac430 2024-05-09T05:00:38.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.609+0000 7f663559a640 1 -- 172.21.15.60:0/2765757555 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6620003520 con 0x7f66380ac430 2024-05-09T05:00:38.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f663559a640 1 -- 172.21.15.60:0/2765757555 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2744756197 0 0) 0x7f6638071e60 con 0x7f6638073c30 2024-05-09T05:00:38.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f663559a640 1 -- 172.21.15.60:0/2765757555 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6638071b80 con 0x7f6638073c30 2024-05-09T05:00:38.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f663559a640 1 -- 172.21.15.60:0/2765757555 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2780311017 0 0) 0x7f6620003520 con 0x7f66380ac430 2024-05-09T05:00:38.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f663559a640 1 -- 172.21.15.60:0/2765757555 >> v2:172.21.15.114:3300/0 conn(0x7f663815c3a0 msgr2=0x7f6638070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:38.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f663559a640 1 --2- 172.21.15.60:0/2765757555 >> v2:172.21.15.114:3300/0 conn(0x7f663815c3a0 0x7f6638070150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f663559a640 1 -- 172.21.15.60:0/2765757555 >> v1:172.21.15.114:6789/0 conn(0x7f6638073c30 legacy=0x7f663809a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:38.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f663559a640 1 -- 172.21.15.60:0/2765757555 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6638072420 con 0x7f66380ac430 2024-05-09T05:00:38.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f663559a640 1 -- 172.21.15.60:0/2765757555 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f662c0016f0 con 0x7f66380ac430 2024-05-09T05:00:38.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f663559a640 1 -- 172.21.15.60:0/2765757555 >> v1:172.21.15.60:6789/0 conn(0x7f66380ac430 legacy=0x7f6638099a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:38.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f663559a640 1 --2- 172.21.15.60:0/2765757555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6620003c30 0x7f6620004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f663559a640 1 --2- 172.21.15.60:0/2765757555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6620004800 0x7f6620004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f663559a640 1 --2- 172.21.15.60:0/2765757555 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6620005210 0x7f662000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f663559a640 1 -- 172.21.15.60:0/2765757555 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6638072140 con 0x7f6620004800 2024-05-09T05:00:38.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f663559a640 1 -- 172.21.15.60:0/2765757555 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f662000c140 con 0x7f6620003c30 2024-05-09T05:00:38.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f663559a640 1 -- 172.21.15.60:0/2765757555 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f662000e550 con 0x7f6620005210 2024-05-09T05:00:38.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f6635d9b640 1 --2- 172.21.15.60:0/2765757555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6620004800 0x7f6620004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f663659c640 1 --2- 172.21.15.60:0/2765757555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6620003c30 0x7f6620004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f6636d9d640 1 --2- 172.21.15.60:0/2765757555 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6620005210 0x7f662000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f6635d9b640 1 -- 172.21.15.60:0/2765757555 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6620005210 msgr2=0x7f662000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:38.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f6635d9b640 1 --2- 172.21.15.60:0/2765757555 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6620005210 0x7f662000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f6635d9b640 1 -- 172.21.15.60:0/2765757555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6620003c30 msgr2=0x7f6620004030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:38.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f6635d9b640 1 --2- 172.21.15.60:0/2765757555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6620003c30 0x7f6620004030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f6635d9b640 1 -- 172.21.15.60:0/2765757555 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6620003520 con 0x7f6620004800 2024-05-09T05:00:38.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.610+0000 7f6636d9d640 1 -- 172.21.15.60:0/2765757555 reap_dead start 2024-05-09T05:00:38.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.611+0000 7f6635d9b640 1 --2- 172.21.15.60:0/2765757555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6620004800 0x7f6620004c80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f662c007fd0 tx=0x7f6620005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1507dc6d1a4a821 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:38.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.611+0000 7f663559a640 1 -- 172.21.15.60:0/2765757555 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c007ce0 con 0x7f6620004800 2024-05-09T05:00:38.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.611+0000 7f663559a640 1 -- 172.21.15.60:0/2765757555 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f662c005af0 con 0x7f6620004800 2024-05-09T05:00:38.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.611+0000 7f663559a640 1 -- 172.21.15.60:0/2765757555 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c007070 con 0x7f6620004800 2024-05-09T05:00:38.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.612+0000 7f663c946640 1 -- 172.21.15.60:0/2765757555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6620004800 msgr2=0x7f6620004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:38.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.612+0000 7f663c946640 1 --2- 172.21.15.60:0/2765757555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6620004800 0x7f6620004c80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f662c007fd0 tx=0x7f6620005830 comp rx=0 tx=0).stop 2024-05-09T05:00:38.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.612+0000 7f663c946640 1 -- 172.21.15.60:0/2765757555 shutdown_connections 2024-05-09T05:00:38.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.612+0000 7f663c946640 1 --2- 172.21.15.60:0/2765757555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6620004800 0x7f6620004c80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.612+0000 7f663c946640 1 -- 172.21.15.60:0/2765757555 >> 172.21.15.60:0/2765757555 conn(0x7f66380972f0 msgr2=0x7f66380976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:38.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.612+0000 7f663c946640 1 -- 172.21.15.60:0/2765757555 shutdown_connections 2024-05-09T05:00:38.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.612+0000 7f663c946640 1 -- 172.21.15.60:0/2765757555 wait complete. 2024-05-09T05:00:38.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.613+0000 7f663c946640 1 Processor -- start 2024-05-09T05:00:38.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.613+0000 7f663c946640 1 -- start start 2024-05-09T05:00:38.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.614+0000 7f663c946640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6620004800 0x7f6638156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.614+0000 7f663c946640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6638070090 0x7f6638070510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.614+0000 7f663c946640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f663808e6c0 0x7f663808eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.614+0000 7f663c946640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6638071810 con 0x7f663808e6c0 2024-05-09T05:00:38.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.614+0000 7f663c946640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f663815c780 con 0x7f6638070090 2024-05-09T05:00:38.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.614+0000 7f663c946640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6638097de0 con 0x7f6620004800 2024-05-09T05:00:38.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.614+0000 7f6636d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f663808e6c0 0x7f663808eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.614+0000 7f6636d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f663808e6c0 0x7f663808eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35244/0 (socket says 172.21.15.60:35244) 2024-05-09T05:00:38.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.614+0000 7f6636d9d640 1 -- 172.21.15.60:0/147631047 learned_addr learned my addr 172.21.15.60:0/147631047 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:38.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.614+0000 7f663659c640 1 --2- 172.21.15.60:0/147631047 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6620004800 0x7f6638156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.614+0000 7f6636d9d640 1 -- 172.21.15.60:0/147631047 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6620004800 msgr2=0x7f6638156940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:38.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.614+0000 7f6636d9d640 1 --2- 172.21.15.60:0/147631047 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6620004800 0x7f6638156940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.614+0000 7f6636d9d640 1 -- 172.21.15.60:0/147631047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6638070090 msgr2=0x7f6638070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:38.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.614+0000 7f6635d9b640 1 --2- 172.21.15.60:0/147631047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6638070090 0x7f6638070510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.614+0000 7f6636d9d640 1 --2- 172.21.15.60:0/147631047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6638070090 0x7f6638070510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.614+0000 7f6636d9d640 1 -- 172.21.15.60:0/147631047 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f663808f3f0 con 0x7f663808e6c0 2024-05-09T05:00:38.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.614+0000 7f6635d9b640 1 --2- 172.21.15.60:0/147631047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6638070090 0x7f6638070510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:38.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.614+0000 7f6636d9d640 1 --2- 172.21.15.60:0/147631047 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f663808e6c0 0x7f663808eb40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f66380b2270 tx=0x7f663000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:38.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.615+0000 7f6626ffd640 1 -- 172.21.15.60:0/147631047 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663000ee40 con 0x7f663808e6c0 2024-05-09T05:00:38.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.615+0000 7f6626ffd640 1 -- 172.21.15.60:0/147631047 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6630003b80 con 0x7f663808e6c0 2024-05-09T05:00:38.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.615+0000 7f663c946640 1 -- 172.21.15.60:0/147631047 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f663808f6d0 con 0x7f663808e6c0 2024-05-09T05:00:38.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.615+0000 7f6626ffd640 1 -- 172.21.15.60:0/147631047 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6630003d70 con 0x7f663808e6c0 2024-05-09T05:00:38.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.615+0000 7f663c946640 1 -- 172.21.15.60:0/147631047 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f663808fc90 con 0x7f663808e6c0 2024-05-09T05:00:38.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.616+0000 7f6626ffd640 1 -- 172.21.15.60:0/147631047 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f663001a020 con 0x7f663808e6c0 2024-05-09T05:00:38.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.617+0000 7f6626ffd640 1 --2- 172.21.15.60:0/147631047 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f66180762d0 0x7f6618078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.617+0000 7f6626ffd640 1 -- 172.21.15.60:0/147631047 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6630096df0 con 0x7f663808e6c0 2024-05-09T05:00:38.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.617+0000 7f663659c640 1 --2- 172.21.15.60:0/147631047 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f66180762d0 0x7f6618078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.618+0000 7f663659c640 1 --2- 172.21.15.60:0/147631047 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f66180762d0 0x7f6618078790 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f66280099a0 tx=0x7f6628008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:38.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.618+0000 7f663f669740 1 Processor -- start 2024-05-09T05:00:38.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.618+0000 7f663f669740 1 -- start start 2024-05-09T05:00:38.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f663f669740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d0ff0 0x560da95d3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f663f669740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95d13d0 0x560da95d3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f663f669740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95a3c50 0x560da95a7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f663f669740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x560da950d8d0 con 0x560da95d0ff0 2024-05-09T05:00:38.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f663f669740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560da94c6c10 con 0x560da95d13d0 2024-05-09T05:00:38.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f663f669740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x560da9509890 con 0x560da95a3c50 2024-05-09T05:00:38.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f663659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d0ff0 0x560da95d3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f663659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d0ff0 0x560da95d3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35254/0 (socket says 172.21.15.60:35254) 2024-05-09T05:00:38.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f663659c640 1 -- 172.21.15.60:0/3983298846 learned_addr learned my addr 172.21.15.60:0/3983298846 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:38.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f6635d9b640 1 --2- 172.21.15.60:0/3983298846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95d13d0 0x560da95d3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f6636d9d640 1 --2- 172.21.15.60:0/3983298846 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95a3c50 0x560da95a7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f663659c640 1 -- 172.21.15.60:0/3983298846 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95a3c50 msgr2=0x560da95a7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f663659c640 1 --2- 172.21.15.60:0/3983298846 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95a3c50 0x560da95a7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f663659c640 1 -- 172.21.15.60:0/3983298846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95d13d0 msgr2=0x560da95d3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f663659c640 1 --2- 172.21.15.60:0/3983298846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95d13d0 0x560da95d3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f663659c640 1 -- 172.21.15.60:0/3983298846 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560da94ce5d0 con 0x560da95d0ff0 2024-05-09T05:00:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f663659c640 1 --2- 172.21.15.60:0/3983298846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d0ff0 0x560da95d3800 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6628013d00 tx=0x7f6628016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8446c15f716a1755 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f6617fff640 1 -- 172.21.15.60:0/3983298846 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6628013d30 con 0x560da95d0ff0 2024-05-09T05:00:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f6617fff640 1 -- 172.21.15.60:0/3983298846 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6628028070 con 0x560da95d0ff0 2024-05-09T05:00:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.619+0000 7f6617fff640 1 -- 172.21.15.60:0/3983298846 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6628023640 con 0x560da95d0ff0 2024-05-09T05:00:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.620+0000 7f663f669740 1 -- 172.21.15.60:0/3983298846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d0ff0 msgr2=0x560da95d3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.620+0000 7f663f669740 1 --2- 172.21.15.60:0/3983298846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d0ff0 0x560da95d3800 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6628013d00 tx=0x7f6628016000 comp rx=0 tx=0).stop 2024-05-09T05:00:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.620+0000 7f663f669740 1 -- 172.21.15.60:0/3983298846 shutdown_connections 2024-05-09T05:00:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.620+0000 7f663f669740 1 --2- 172.21.15.60:0/3983298846 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95a3c50 0x560da95a7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.620+0000 7f663f669740 1 --2- 172.21.15.60:0/3983298846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95d13d0 0x560da95d3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.620+0000 7f663f669740 1 --2- 172.21.15.60:0/3983298846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d0ff0 0x560da95d3800 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.620+0000 7f663f669740 1 -- 172.21.15.60:0/3983298846 >> 172.21.15.60:0/3983298846 conn(0x560da95a5ec0 msgr2=0x560da95ee800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:38.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.620+0000 7f663f669740 1 -- 172.21.15.60:0/3983298846 shutdown_connections 2024-05-09T05:00:38.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.620+0000 7f663f669740 1 -- 172.21.15.60:0/3983298846 wait complete. 2024-05-09T05:00:38.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.621+0000 7f663f669740 1 -- start start 2024-05-09T05:00:38.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.621+0000 7f663f669740 1 Processor -- start 2024-05-09T05:00:38.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.622+0000 7f663f669740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95d0ff0 0x560da95e7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.622+0000 7f663f669740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 0x560da95ef930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.622+0000 7f663f669740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95e75b0 0x560da95efe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.622+0000 7f663f669740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x560da95a9070 con 0x560da95d13d0 2024-05-09T05:00:38.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.622+0000 7f663f669740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560da94ca760 con 0x560da95e75b0 2024-05-09T05:00:38.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.622+0000 7f663f669740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x560da95a9350 con 0x560da95d0ff0 2024-05-09T05:00:38.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.622+0000 7f663f669740 10 client.0 _ll_register_callbacks cb 0x560da9503cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:00:38.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.622+0000 7f6635d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 0x560da95ef930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.622+0000 7f6635d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 0x560da95ef930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35262/0 (socket says 172.21.15.60:35262) 2024-05-09T05:00:38.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.622+0000 7f6635d9b640 1 -- 172.21.15.60:0/3573666355 learned_addr learned my addr 172.21.15.60:0/3573666355 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:38.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.622+0000 7f6636d9d640 1 --2- 172.21.15.60:0/3573666355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95e75b0 0x560da95efe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.622+0000 7f663659c640 1 --2- 172.21.15.60:0/3573666355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95d0ff0 0x560da95e7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.622+0000 7f6635d9b640 1 -- 172.21.15.60:0/3573666355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95d0ff0 msgr2=0x560da95e7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:38.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.622+0000 7f6635d9b640 1 --2- 172.21.15.60:0/3573666355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95d0ff0 0x560da95e7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.622+0000 7f6635d9b640 1 -- 172.21.15.60:0/3573666355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95e75b0 msgr2=0x560da95efe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:38.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.622+0000 7f6635d9b640 1 --2- 172.21.15.60:0/3573666355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95e75b0 0x560da95efe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.622+0000 7f6635d9b640 1 -- 172.21.15.60:0/3573666355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560da921f2d0 con 0x560da95d13d0 2024-05-09T05:00:38.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.622+0000 7f6635d9b640 1 --2- 172.21.15.60:0/3573666355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 0x560da95ef930 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f662c02acf0 tx=0x7f662c02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db86e9e57f0861 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:38.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.623+0000 7f66157fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-05-09T05:00:38.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.623+0000 7f66157fa640 1 -- 172.21.15.60:0/3573666355 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c037040 con 0x560da95d13d0 2024-05-09T05:00:38.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.623+0000 7f66157fa640 1 -- 172.21.15.60:0/3573666355 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f662c0047e0 con 0x560da95d13d0 2024-05-09T05:00:38.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.623+0000 7f663f669740 1 -- 172.21.15.60:0/3573666355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6628028070 con 0x560da95d13d0 2024-05-09T05:00:38.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.623+0000 7f66157fa640 1 -- 172.21.15.60:0/3573666355 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f662c04b050 con 0x560da95d13d0 2024-05-09T05:00:38.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.623+0000 7f663f669740 10 client.4668 fetch_fsmap learned FSMap version 9 2024-05-09T05:00:38.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.623+0000 7f663f669740 1 -- 172.21.15.60:0/3573666355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x560da92e96b0 con 0x560da95d13d0 2024-05-09T05:00:38.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.623+0000 7f66157fa640 1 -- 172.21.15.60:0/3573666355 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6628028070 con 0x560da95d13d0 2024-05-09T05:00:38.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.623+0000 7f66157fa640 1 -- 172.21.15.60:0/3573666355 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7f662c04e020 con 0x560da95d13d0 2024-05-09T05:00:38.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.623+0000 7f663f669740 10 client.4668 fetch_fsmap finished waiting for FSMap version 9 2024-05-09T05:00:38.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.623+0000 7f663f669740 10 client.4668 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T05:00:38.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.623+0000 7f663f669740 20 client.4668 populate_metadata read hostname 'smithi060' 2024-05-09T05:00:38.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.623+0000 7f663f669740 1 --2- 172.21.15.60:0/3573666355 >> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] conn(0x560da9602890 0x560da95e94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.623+0000 7f663f669740 4 client.4668 mds_command: new command op to 4261 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T05:00:38.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.623+0000 7f663f669740 1 -- 172.21.15.60:0/3573666355 --> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560da9491760 con 0x560da9602890 2024-05-09T05:00:38.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.623+0000 7f663659c640 1 --2- 172.21.15.60:0/3573666355 >> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] conn(0x560da9602890 0x560da95e94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.624+0000 7f66157fa640 10 client.4668 ms_handle_connect on v2:172.21.15.60:6834/2135423314 2024-05-09T05:00:38.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.624+0000 7f663659c640 1 --2- 172.21.15.60:0/3573666355 >> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] conn(0x560da9602890 0x560da95e94d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d697f0b5c81c2338 server_cookie=9228d566a5038498 in_seq=0 out_seq=0 2024-05-09T05:00:38.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.624+0000 7f66157fa640 1 -- 172.21.15.60:0/3573666355 <== mds.0 v2:172.21.15.60:6834/2135423314 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560da9491760 con 0x560da9602890 2024-05-09T05:00:38.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.624+0000 7f66157fa640 10 client.4668 handle_command_reply: tid=0 2024-05-09T05:00:38.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.624+0000 7f663f669740 1 client.4668 shutdown 2024-05-09T05:00:38.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.625+0000 7f663f669740 1 -- 172.21.15.60:0/3573666355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 msgr2=0x560da95ef930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:38.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.625+0000 7f663f669740 1 --2- 172.21.15.60:0/3573666355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 0x560da95ef930 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f662c02acf0 tx=0x7f662c02c000 comp rx=0 tx=0).stop 2024-05-09T05:00:38.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.625+0000 7f663f669740 1 -- 172.21.15.60:0/3573666355 shutdown_connections 2024-05-09T05:00:38.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.625+0000 7f663f669740 1 --2- 172.21.15.60:0/3573666355 >> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] conn(0x560da9602890 0x560da95e94d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.625+0000 7f663f669740 1 --2- 172.21.15.60:0/3573666355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95e75b0 0x560da95efe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.625+0000 7f663f669740 1 --2- 172.21.15.60:0/3573666355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 0x560da95ef930 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.625+0000 7f66157fa640 0 client.4668 ms_handle_reset on v2:172.21.15.60:6834/2135423314 2024-05-09T05:00:38.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.625+0000 7f663f669740 1 --2- 172.21.15.60:0/3573666355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95d0ff0 0x560da95e7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.625+0000 7f663f669740 1 -- 172.21.15.60:0/3573666355 >> 172.21.15.60:0/3573666355 conn(0x560da95a3c50 msgr2=0x560da95a2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:38.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.625+0000 7f663f669740 1 -- 172.21.15.60:0/3573666355 shutdown_connections 2024-05-09T05:00:38.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.625+0000 7f663f669740 1 -- 172.21.15.60:0/3573666355 wait complete. 2024-05-09T05:00:38.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.625+0000 7f663f669740 20 client.4668 trim_cache size 0 max 16384 2024-05-09T05:00:38.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.644+0000 7f663f669740 1 Processor -- start 2024-05-09T05:00:38.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.644+0000 7f663f669740 1 -- start start 2024-05-09T05:00:38.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.644+0000 7f663f669740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95d0ff0 0x560da95f1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.644+0000 7f663f669740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 0x560da9613380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.644+0000 7f663f669740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95e75b0 0x560da95a7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.644+0000 7f663f669740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6638202670 con 0x560da95d13d0 2024-05-09T05:00:38.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.644+0000 7f663f669740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f66382019d0 con 0x560da95e75b0 2024-05-09T05:00:38.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.644+0000 7f663f669740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6638202020 con 0x560da95d0ff0 2024-05-09T05:00:38.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.644+0000 7f6635d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 0x560da9613380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.645+0000 7f6635d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 0x560da9613380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35264/0 (socket says 172.21.15.60:35264) 2024-05-09T05:00:38.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.645+0000 7f6635d9b640 1 -- 172.21.15.60:0/547817641 learned_addr learned my addr 172.21.15.60:0/547817641 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:38.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.645+0000 7f6636d9d640 1 --2- 172.21.15.60:0/547817641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95e75b0 0x560da95a7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.645+0000 7f663659c640 1 --2- 172.21.15.60:0/547817641 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95d0ff0 0x560da95f1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.645+0000 7f6635d9b640 1 -- 172.21.15.60:0/547817641 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95d0ff0 msgr2=0x560da95f1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:38.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.645+0000 7f6635d9b640 1 --2- 172.21.15.60:0/547817641 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95d0ff0 0x560da95f1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.645+0000 7f6635d9b640 1 -- 172.21.15.60:0/547817641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95e75b0 msgr2=0x560da95a7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:38.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.645+0000 7f6635d9b640 1 --2- 172.21.15.60:0/547817641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95e75b0 0x560da95a7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.645+0000 7f6635d9b640 1 -- 172.21.15.60:0/547817641 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560da95a8d30 con 0x560da95d13d0 2024-05-09T05:00:38.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.645+0000 7f6635d9b640 1 --2- 172.21.15.60:0/547817641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 0x560da9613380 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f662c000ef0 tx=0x7f662c03e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dd4972208cbcebb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:38.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.645+0000 7f66177fe640 1 -- 172.21.15.60:0/547817641 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c001700 con 0x560da95d13d0 2024-05-09T05:00:38.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.645+0000 7f66177fe640 1 -- 172.21.15.60:0/547817641 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f662c038040 con 0x560da95d13d0 2024-05-09T05:00:38.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.645+0000 7f66177fe640 1 -- 172.21.15.60:0/547817641 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c007070 con 0x560da95d13d0 2024-05-09T05:00:38.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.646+0000 7f663f669740 1 -- 172.21.15.60:0/547817641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 msgr2=0x560da9613380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:38.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.646+0000 7f663f669740 1 --2- 172.21.15.60:0/547817641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 0x560da9613380 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f662c000ef0 tx=0x7f662c03e000 comp rx=0 tx=0).stop 2024-05-09T05:00:38.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.646+0000 7f663f669740 1 -- 172.21.15.60:0/547817641 shutdown_connections 2024-05-09T05:00:38.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.646+0000 7f663f669740 1 --2- 172.21.15.60:0/547817641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95e75b0 0x560da95a7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.646+0000 7f663f669740 1 --2- 172.21.15.60:0/547817641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 0x560da9613380 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.646+0000 7f663f669740 1 --2- 172.21.15.60:0/547817641 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95d0ff0 0x560da95f1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.646+0000 7f663f669740 1 -- 172.21.15.60:0/547817641 >> 172.21.15.60:0/547817641 conn(0x560da95a3c50 msgr2=0x560da95e6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:38.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.646+0000 7f663f669740 1 -- 172.21.15.60:0/547817641 shutdown_connections 2024-05-09T05:00:38.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.646+0000 7f663f669740 1 -- 172.21.15.60:0/547817641 wait complete. 2024-05-09T05:00:38.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.646+0000 7f663f669740 1 -- start start 2024-05-09T05:00:38.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.646+0000 7f663f669740 1 Processor -- start 2024-05-09T05:00:38.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.647+0000 7f663f669740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95d0ff0 0x560da95a3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.647+0000 7f663f669740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 0x560da95e4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.647+0000 7f663f669740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95e75b0 0x560da95a7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.647+0000 7f663f669740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x560da9602710 con 0x560da95d13d0 2024-05-09T05:00:38.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.647+0000 7f663f669740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560da95f2740 con 0x560da95e75b0 2024-05-09T05:00:38.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.647+0000 7f663f669740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x560da9602cd0 con 0x560da95d0ff0 2024-05-09T05:00:38.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.647+0000 7f663f669740 10 client.0 _ll_register_callbacks cb 0x7f6620003880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:00:38.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.647+0000 7f6635d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 0x560da95e4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.647+0000 7f6635d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 0x560da95e4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35280/0 (socket says 172.21.15.60:35280) 2024-05-09T05:00:38.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.647+0000 7f6635d9b640 1 -- 172.21.15.60:0/1188532422 learned_addr learned my addr 172.21.15.60:0/1188532422 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:38.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.648+0000 7f663659c640 1 --2- 172.21.15.60:0/1188532422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95d0ff0 0x560da95a3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.648+0000 7f6636d9d640 1 --2- 172.21.15.60:0/1188532422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95e75b0 0x560da95a7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.648+0000 7f6635d9b640 1 -- 172.21.15.60:0/1188532422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95d0ff0 msgr2=0x560da95a3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:38.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.648+0000 7f6635d9b640 1 --2- 172.21.15.60:0/1188532422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95d0ff0 0x560da95a3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.648+0000 7f6635d9b640 1 -- 172.21.15.60:0/1188532422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95e75b0 msgr2=0x560da95a7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:38.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.648+0000 7f6635d9b640 1 --2- 172.21.15.60:0/1188532422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95e75b0 0x560da95a7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.648+0000 7f6635d9b640 1 -- 172.21.15.60:0/1188532422 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560da95f01e0 con 0x560da95d13d0 2024-05-09T05:00:38.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.648+0000 7f6635d9b640 1 --2- 172.21.15.60:0/1188532422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 0x560da95e4ba0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f662c008fb0 tx=0x7f662c007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3741b25c97816d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:38.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.648+0000 7f66157fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-05-09T05:00:38.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.648+0000 7f66157fa640 1 -- 172.21.15.60:0/1188532422 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c004c30 con 0x560da95d13d0 2024-05-09T05:00:38.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.648+0000 7f66157fa640 1 -- 172.21.15.60:0/1188532422 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f662c007670 con 0x560da95d13d0 2024-05-09T05:00:38.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.648+0000 7f663f669740 1 -- 172.21.15.60:0/1188532422 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f662c038040 con 0x560da95d13d0 2024-05-09T05:00:38.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.648+0000 7f66157fa640 1 -- 172.21.15.60:0/1188532422 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f662c063050 con 0x560da95d13d0 2024-05-09T05:00:38.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.648+0000 7f663f669740 10 client.4674 fetch_fsmap learned FSMap version 9 2024-05-09T05:00:38.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.648+0000 7f663f669740 1 -- 172.21.15.60:0/1188532422 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x560da95a2c10 con 0x560da95d13d0 2024-05-09T05:00:38.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.648+0000 7f66157fa640 1 -- 172.21.15.60:0/1188532422 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f662c038040 con 0x560da95d13d0 2024-05-09T05:00:38.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.648+0000 7f66157fa640 1 -- 172.21.15.60:0/1188532422 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7f662c066020 con 0x560da95d13d0 2024-05-09T05:00:38.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.648+0000 7f663f669740 10 client.4674 fetch_fsmap finished waiting for FSMap version 9 2024-05-09T05:00:38.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f663f669740 10 client.4674 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T05:00:38.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f663f669740 20 client.4674 populate_metadata read hostname 'smithi060' 2024-05-09T05:00:38.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f663f669740 1 --2- 172.21.15.60:0/1188532422 >> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] conn(0x560da966dd60 0x560da95ee800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:38.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f663f669740 4 client.4674 mds_command: new command op to 4261 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T05:00:38.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f663f669740 1 -- 172.21.15.60:0/1188532422 --> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] -- command(tid 0: {"prefix": "status"}) -- 0x560da951b4e0 con 0x560da966dd60 2024-05-09T05:00:38.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f663659c640 1 --2- 172.21.15.60:0/1188532422 >> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] conn(0x560da966dd60 0x560da95ee800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:38.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f66157fa640 10 client.4674 ms_handle_connect on v2:172.21.15.60:6834/2135423314 2024-05-09T05:00:38.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f663659c640 1 --2- 172.21.15.60:0/1188532422 >> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] conn(0x560da966dd60 0x560da95ee800 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85cfdab00b01ea9 server_cookie=d1367aa35a8484da in_seq=0 out_seq=0 2024-05-09T05:00:38.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f66157fa640 1 -- 172.21.15.60:0/1188532422 <== mds.0 v2:172.21.15.60:6834/2135423314 1 ==== command_reply(tid 0: 0 ) ==== 8+0+323 (crc 0 0 0) 0x7f6628039e50 con 0x560da966dd60 2024-05-09T05:00:38.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f66157fa640 10 client.4674 handle_command_reply: tid=0 2024-05-09T05:00:38.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f663f669740 1 client.4674 shutdown 2024-05-09T05:00:38.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f663f669740 1 -- 172.21.15.60:0/1188532422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 msgr2=0x560da95e4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:38.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f663f669740 1 --2- 172.21.15.60:0/1188532422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 0x560da95e4ba0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f662c008fb0 tx=0x7f662c007e80 comp rx=0 tx=0).stop 2024-05-09T05:00:38.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f663f669740 1 -- 172.21.15.60:0/1188532422 shutdown_connections 2024-05-09T05:00:38.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f663f669740 1 --2- 172.21.15.60:0/1188532422 >> [v2:172.21.15.60:6834/2135423314,v1:172.21.15.60:6835/2135423314] conn(0x560da966dd60 0x560da95ee800 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f663f669740 1 --2- 172.21.15.60:0/1188532422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560da95e75b0 0x560da95a7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f66157fa640 0 client.4674 ms_handle_reset on v2:172.21.15.60:6834/2135423314 2024-05-09T05:00:38.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f663f669740 1 --2- 172.21.15.60:0/1188532422 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560da95d13d0 0x560da95e4ba0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f663f669740 1 --2- 172.21.15.60:0/1188532422 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560da95d0ff0 0x560da95a3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.649+0000 7f663f669740 1 -- 172.21.15.60:0/1188532422 >> 172.21.15.60:0/1188532422 conn(0x560da95a3c50 msgr2=0x560da95e5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:38.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.650+0000 7f663f669740 1 -- 172.21.15.60:0/1188532422 shutdown_connections 2024-05-09T05:00:38.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.650+0000 7f663f669740 1 -- 172.21.15.60:0/1188532422 wait complete. 2024-05-09T05:00:38.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.650+0000 7f663f669740 20 client.4674 trim_cache size 0 max 16384 2024-05-09T05:00:38.656 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:00:38.657 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2", 2024-05-09T05:00:38.657 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2024-05-09T05:00:38.657 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4261, 2024-05-09T05:00:38.657 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2024-05-09T05:00:38.657 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2024-05-09T05:00:38.657 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2024-05-09T05:00:38.657 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 25.824294161000001, 2024-05-09T05:00:38.657 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 9, 2024-05-09T05:00:38.657 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 24, 2024-05-09T05:00:38.657 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 24, 2024-05-09T05:00:38.657 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 31.57189412 2024-05-09T05:00:38.657 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:00:38.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.650+0000 7f663c946640 1 -- 172.21.15.60:0/147631047 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f66180762d0 msgr2=0x7f6618078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:38.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.650+0000 7f663c946640 1 --2- 172.21.15.60:0/147631047 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f66180762d0 0x7f6618078790 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f66280099a0 tx=0x7f6628008040 comp rx=0 tx=0).stop 2024-05-09T05:00:38.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.650+0000 7f663c946640 1 -- 172.21.15.60:0/147631047 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f663808e6c0 msgr2=0x7f663808eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:38.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.650+0000 7f663c946640 1 --2- 172.21.15.60:0/147631047 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f663808e6c0 0x7f663808eb40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f66380b2270 tx=0x7f663000eb90 comp rx=0 tx=0).stop 2024-05-09T05:00:38.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.650+0000 7f663c946640 1 -- 172.21.15.60:0/147631047 shutdown_connections 2024-05-09T05:00:38.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.650+0000 7f663c946640 1 --2- 172.21.15.60:0/147631047 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f66180762d0 0x7f6618078790 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.650+0000 7f663c946640 1 --2- 172.21.15.60:0/147631047 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f663808e6c0 0x7f663808eb40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.650+0000 7f663c946640 1 --2- 172.21.15.60:0/147631047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6638070090 0x7f6638070510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.650+0000 7f663c946640 1 --2- 172.21.15.60:0/147631047 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6620004800 0x7f6638156940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:38.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.650+0000 7f663c946640 1 -- 172.21.15.60:0/147631047 >> 172.21.15.60:0/147631047 conn(0x7f66380972f0 msgr2=0x7f66380b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:38.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.650+0000 7f663c946640 1 -- 172.21.15.60:0/147631047 shutdown_connections 2024-05-09T05:00:38.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:38.650+0000 7f663c946640 1 -- 172.21.15.60:0/147631047 wait complete. 2024-05-09T05:00:38.662 INFO:teuthology.run_tasks:Running task sequential... 2024-05-09T05:00:38.671 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-09T05:00:38.671 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-09T05:00:38.672 INFO:teuthology.task.print:Enabling mgr modules 2024-05-09T05:00:38.672 INFO:teuthology.run_tasks:Running task kclient... 2024-05-09T05:00:38.701 INFO:tasks.kclient:Mounting kernel clients... 2024-05-09T05:00:38.701 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-05-09T05:00:38.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:38.702 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2024-05-09T05:00:38.718 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-05-09T05:00:38.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:38.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T05:00:38.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:38.784 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link delete ceph-brx 2024-05-09T05:00:38.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:38.881 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2024-05-09T05:00:38.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:38.897 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link delete ceph-brx 2024-05-09T05:00:38.966 INFO:teuthology.orchestra.run.smithi060.stderr:Cannot find device "ceph-brx" 2024-05-09T05:00:38.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T05:00:38.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:38.970 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-09T05:00:38.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:38.990 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2024-05-09T05:00:39.062 INFO:teuthology.orchestra.run.smithi114.stderr:Cannot find device "ceph-brx" 2024-05-09T05:00:39.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T05:00:39.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:39.064 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-09T05:00:39.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:39.123 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2024-05-09T05:00:39.192 INFO:teuthology.orchestra.run.smithi114.stderr:Cannot find device "ceph-brx" 2024-05-09T05:00:39.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T05:00:39.194 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-05-09T05:00:39.194 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T05:00:39.194 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T05:00:39.194 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T05:00:39.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.299+0000 7f2f48f58640 1 Processor -- start 2024-05-09T05:00:39.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.299+0000 7f2f48f58640 1 -- start start 2024-05-09T05:00:39.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.300+0000 7f2f48f58640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2f44072540 0x7f2f44097de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:39.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.300+0000 7f2f48f58640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2f4406f750 0x7f2f44072e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:39.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.300+0000 7f2f48f58640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f4415cc30 con 0x7f2f4415b760 2024-05-09T05:00:39.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.300+0000 7f2f48f58640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f2f44071810 con 0x7f2f4406f750 2024-05-09T05:00:39.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.300+0000 7f2f48f58640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f2f440720b0 con 0x7f2f44072540 2024-05-09T05:00:39.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.300+0000 7f2f4259c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2f4415b760 0x7f2f4406f040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33198/0 (socket says 172.21.15.60:33198) 2024-05-09T05:00:39.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.300+0000 7f2f4259c640 1 -- 172.21.15.60:0/2298680487 learned_addr learned my addr 172.21.15.60:0/2298680487 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:39.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.300+0000 7f2f42d9d640 1 --2- 172.21.15.60:0/2298680487 >> v2:172.21.15.60:3300/0 conn(0x7f2f4406f750 0x7f2f44072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:39.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.300+0000 7f2f39d9b640 1 --2- 172.21.15.60:0/2298680487 >> v2:172.21.15.114:3300/0 conn(0x7f2f44072540 0x7f2f44097de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:39.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.300+0000 7f2f42d9d640 1 -- 172.21.15.60:0/2298680487 >> v2:172.21.15.114:3300/0 conn(0x7f2f44072540 msgr2=0x7f2f44097de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:39.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.300+0000 7f2f42d9d640 1 --2- 172.21.15.60:0/2298680487 >> v2:172.21.15.114:3300/0 conn(0x7f2f44072540 0x7f2f44097de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:39.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.300+0000 7f2f42d9d640 1 -- 172.21.15.60:0/2298680487 >> v1:172.21.15.60:6789/0 conn(0x7f2f4415b760 legacy=0x7f2f4406f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:39.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.300+0000 7f2f42d9d640 1 -- 172.21.15.60:0/2298680487 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f44071a60 con 0x7f2f4406f750 2024-05-09T05:00:39.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.300+0000 7f2f41d9b640 1 -- 172.21.15.60:0/2298680487 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1122439882 0 0) 0x7f2f4415cc30 con 0x7f2f4415b760 2024-05-09T05:00:39.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.300+0000 7f2f39d9b640 1 --2- 172.21.15.60:0/2298680487 >> v2:172.21.15.114:3300/0 conn(0x7f2f44072540 0x7f2f44097de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:39.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.301+0000 7f2f42d9d640 1 --2- 172.21.15.60:0/2298680487 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f4406f750 0x7f2f44072e80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2f440752e0 tx=0x7f2f3c02b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8d15b532e4e860 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:39.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.301+0000 7f2f41d9b640 1 -- 172.21.15.60:0/2298680487 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c008640 con 0x7f2f4406f750 2024-05-09T05:00:39.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.301+0000 7f2f41d9b640 1 -- 172.21.15.60:0/2298680487 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f3c0087e0 con 0x7f2f4406f750 2024-05-09T05:00:39.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.301+0000 7f2f41d9b640 1 -- 172.21.15.60:0/2298680487 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c008bb0 con 0x7f2f4406f750 2024-05-09T05:00:39.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.301+0000 7f2f48f58640 1 -- 172.21.15.60:0/2298680487 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f4406f750 msgr2=0x7f2f44072e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:39.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.301+0000 7f2f48f58640 1 --2- 172.21.15.60:0/2298680487 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f4406f750 0x7f2f44072e80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2f440752e0 tx=0x7f2f3c02b780 comp rx=0 tx=0).stop 2024-05-09T05:00:39.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.302+0000 7f2f48f58640 1 -- 172.21.15.60:0/2298680487 shutdown_connections 2024-05-09T05:00:39.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.302+0000 7f2f48f58640 1 --2- 172.21.15.60:0/2298680487 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f4406f750 0x7f2f44072e80 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:39.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.302+0000 7f2f48f58640 1 --2- 172.21.15.60:0/2298680487 >> v2:172.21.15.114:3300/0 conn(0x7f2f44072540 0x7f2f44097de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:39.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.302+0000 7f2f48f58640 1 -- 172.21.15.60:0/2298680487 >> 172.21.15.60:0/2298680487 conn(0x7f2f440972f0 msgr2=0x7f2f440976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:39.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.302+0000 7f2f48f58640 1 -- 172.21.15.60:0/2298680487 shutdown_connections 2024-05-09T05:00:39.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.302+0000 7f2f48f58640 1 -- 172.21.15.60:0/2298680487 wait complete. 2024-05-09T05:00:39.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.303+0000 7f2f48f58640 1 Processor -- start 2024-05-09T05:00:39.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.303+0000 7f2f48f58640 1 -- start start 2024-05-09T05:00:39.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.304+0000 7f2f48f58640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f4406f750 0x7f2f44157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:39.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.304+0000 7f2f48f58640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f44072540 0x7f2f4415f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:39.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.304+0000 7f2f48f58640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f4415b760 0x7f2f4415fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:39.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.304+0000 7f2f48f58640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4415ca10 con 0x7f2f4406f750 2024-05-09T05:00:39.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.304+0000 7f2f48f58640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4415c890 con 0x7f2f4415b760 2024-05-09T05:00:39.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.304+0000 7f2f48f58640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f2f4415bb40 con 0x7f2f44072540 2024-05-09T05:00:39.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.304+0000 7f2f4259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f4406f750 0x7f2f44157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:39.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.304+0000 7f2f4259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f4406f750 0x7f2f44157f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45950/0 (socket says 172.21.15.60:45950) 2024-05-09T05:00:39.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.304+0000 7f2f4259c640 1 -- 172.21.15.60:0/4108964583 learned_addr learned my addr 172.21.15.60:0/4108964583 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:39.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.304+0000 7f2f42d9d640 1 --2- 172.21.15.60:0/4108964583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f4415b760 0x7f2f4415fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:39.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.304+0000 7f2f4259c640 1 -- 172.21.15.60:0/4108964583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f44072540 msgr2=0x7f2f4415f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:39.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.304+0000 7f2f39d9b640 1 --2- 172.21.15.60:0/4108964583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f44072540 0x7f2f4415f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:39.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.304+0000 7f2f4259c640 1 --2- 172.21.15.60:0/4108964583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f44072540 0x7f2f4415f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:39.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.304+0000 7f2f4259c640 1 -- 172.21.15.60:0/4108964583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f4415b760 msgr2=0x7f2f4415fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:39.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.304+0000 7f2f4259c640 1 --2- 172.21.15.60:0/4108964583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f4415b760 0x7f2f4415fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:39.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.304+0000 7f2f4259c640 1 -- 172.21.15.60:0/4108964583 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f44158450 con 0x7f2f4406f750 2024-05-09T05:00:39.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.304+0000 7f2f42d9d640 1 --2- 172.21.15.60:0/4108964583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f4415b760 0x7f2f4415fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:39.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.305+0000 7f2f4259c640 1 --2- 172.21.15.60:0/4108964583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f4406f750 0x7f2f44157f10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2f3400c490 tx=0x7f2f3400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:39.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.305+0000 7f2f3b7fe640 1 -- 172.21.15.60:0/4108964583 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f34018070 con 0x7f2f4406f750 2024-05-09T05:00:39.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.305+0000 7f2f3b7fe640 1 -- 172.21.15.60:0/4108964583 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f34007500 con 0x7f2f4406f750 2024-05-09T05:00:39.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.305+0000 7f2f48f58640 1 -- 172.21.15.60:0/4108964583 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f440a94d0 con 0x7f2f4406f750 2024-05-09T05:00:39.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.305+0000 7f2f3b7fe640 1 -- 172.21.15.60:0/4108964583 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f34004a20 con 0x7f2f4406f750 2024-05-09T05:00:39.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.305+0000 7f2f48f58640 1 -- 172.21.15.60:0/4108964583 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f440a9a90 con 0x7f2f4406f750 2024-05-09T05:00:39.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.306+0000 7f2f48f58640 1 -- 172.21.15.60:0/4108964583 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f44051040 con 0x7f2f4406f750 2024-05-09T05:00:39.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.306+0000 7f2f3b7fe640 1 -- 172.21.15.60:0/4108964583 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2f3401b070 con 0x7f2f4406f750 2024-05-09T05:00:39.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.307+0000 7f2f3b7fe640 1 --2- 172.21.15.60:0/4108964583 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2f240762d0 0x7f2f24078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:39.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.307+0000 7f2f3b7fe640 1 -- 172.21.15.60:0/4108964583 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2f34096bf0 con 0x7f2f4406f750 2024-05-09T05:00:39.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.307+0000 7f2f39d9b640 1 --2- 172.21.15.60:0/4108964583 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2f240762d0 0x7f2f24078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:39.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.308+0000 7f2f39d9b640 1 --2- 172.21.15.60:0/4108964583 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2f240762d0 0x7f2f24078790 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2f4415d850 tx=0x7f2f2c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:39.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.312+0000 7f2f3b7fe640 1 -- 172.21.15.60:0/4108964583 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2f44158450 con 0x7f2f4406f750 2024-05-09T05:00:39.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.540+0000 7f2f48f58640 1 -- 172.21.15.60:0/4108964583 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2f44073d90 con 0x7f2f4406f750 2024-05-09T05:00:39.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.540+0000 7f2f3b7fe640 1 -- 172.21.15.60:0/4108964583 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f2f34068070 con 0x7f2f4406f750 2024-05-09T05:00:39.541 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-09T05:00:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.542+0000 7f2f48f58640 1 -- 172.21.15.60:0/4108964583 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2f240762d0 msgr2=0x7f2f24078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.542+0000 7f2f48f58640 1 --2- 172.21.15.60:0/4108964583 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2f240762d0 0x7f2f24078790 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2f4415d850 tx=0x7f2f2c008040 comp rx=0 tx=0).stop 2024-05-09T05:00:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.542+0000 7f2f48f58640 1 -- 172.21.15.60:0/4108964583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f4406f750 msgr2=0x7f2f44157f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.542+0000 7f2f48f58640 1 --2- 172.21.15.60:0/4108964583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f4406f750 0x7f2f44157f10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2f3400c490 tx=0x7f2f3400c950 comp rx=0 tx=0).stop 2024-05-09T05:00:39.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.542+0000 7f2f48f58640 1 -- 172.21.15.60:0/4108964583 shutdown_connections 2024-05-09T05:00:39.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.542+0000 7f2f48f58640 1 --2- 172.21.15.60:0/4108964583 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2f240762d0 0x7f2f24078790 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:39.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.542+0000 7f2f48f58640 1 --2- 172.21.15.60:0/4108964583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f4415b760 0x7f2f4415fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:39.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.542+0000 7f2f48f58640 1 --2- 172.21.15.60:0/4108964583 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f44072540 0x7f2f4415f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:39.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.542+0000 7f2f48f58640 1 --2- 172.21.15.60:0/4108964583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f4406f750 0x7f2f44157f10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:39.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.542+0000 7f2f48f58640 1 -- 172.21.15.60:0/4108964583 >> 172.21.15.60:0/4108964583 conn(0x7f2f440972f0 msgr2=0x7f2f440a8480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:39.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.542+0000 7f2f48f58640 1 -- 172.21.15.60:0/4108964583 shutdown_connections 2024-05-09T05:00:39.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39.542+0000 7f2f48f58640 1 -- 172.21.15.60:0/4108964583 wait complete. 2024-05-09T05:00:39.554 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T05:00:39.554 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T05:00:39.554 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2024-05-09T05:00:39.554 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T05:00:39.554 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T05:00:39.554 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T05:00:39.554 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T05:00:39.554 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T05:00:39.554 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T05:00:39.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:39.555 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2024-05-09T05:00:39.571 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T05:00:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T05:00:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T05:00:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2024-05-09T05:00:39.571 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:39.572 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:00:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2024-05-09T05:00:39.572 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:00:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2024-05-09T05:00:39.572 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T05:00:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2024-05-09T05:00:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T05:00:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 42724sec preferred_lft 42724sec 2024-05-09T05:00:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2024-05-09T05:00:39.572 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:39.573 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:00:39.573 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:39.573 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2024-05-09T05:00:39.573 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-09T05:00:39.573 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:39.573 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-05-09T05:00:39.573 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link add name ceph-brx type bridge 2024-05-09T05:00:39.573 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip addr flush dev ceph-brx 2024-05-09T05:00:39.573 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set ceph-brx up 2024-05-09T05:00:39.573 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-09T05:00:39.574 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-05-09T05:00:39.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:39.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:39.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:39.715 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T05:00:39.740 INFO:teuthology.orchestra.run.smithi060.stdout:1 2024-05-09T05:00:39.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:39.742 DEBUG:teuthology.orchestra.run.smithi060:> route 2024-05-09T05:00:39.800 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2024-05-09T05:00:39.800 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T05:00:39.800 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T05:00:39.800 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T05:00:39.800 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T05:00:39.800 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:39.800 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-05-09T05:00:39.800 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T05:00:39.801 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T05:00:39.801 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T05:00:39.801 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-05-09T05:00:39.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:39.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:39.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:39.930 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2024-05-09T05:00:39.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:39.947 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list-id 2024-05-09T05:00:40.003 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:40.003 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-05-09T05:00:40.003 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T05:00:40.003 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-09T05:00:40.003 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-05-09T05:00:40.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:40.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:40.123 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-09T05:00:40.124 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:40.124 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-05-09T05:00:40.124 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-09T05:00:40.124 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-09T05:00:40.124 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-09T05:00:40.124 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-09T05:00:40.125 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-09T05:00:40.125 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-05-09T05:00:40.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:40.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:40.242 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:40.242 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-05-09T05:00:40.243 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2024-05-09T05:00:40.243 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set dev brx.0 master ceph-brx 2024-05-09T05:00:40.243 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-05-09T05:00:40.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:40.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:40.323 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T05:00:40.323 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T05:00:40.339 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T05:00:40.340 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T05:00:40.340 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T05:00:40.395 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-05-09T05:00:40.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:40.395 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T05:00:40.480 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T05:00:40.480 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T05:00:40.591 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T05:00:40.591 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T05:00:40.591 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.114:3300,172.21.15.114:3301" 2024-05-09T05:00:40.592 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2.cephfs=/ 2024-05-09T05:00:40.592 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.60:3300/172.21.15.114:3300/172.21.15.114:3301" will pass to kernel 2024-05-09T05:00:40.592 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-09T05:00:40.593 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2024-05-09T05:00:40.593 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-09T05:00:40.593 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2024-05-09T05:00:40.593 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-09T05:00:40.594 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-09T05:00:40.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:40.594 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T05:00:40.620 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T05:00:40.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T05:00:40.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:40.719 INFO:teuthology.orchestra.run.smithi060.stdout:client4463 2024-05-09T05:00:40.719 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:00:40.719 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:00:40.736 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T05:00:40.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T05:00:40.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:40.834 INFO:teuthology.orchestra.run.smithi060.stdout:client4463 2024-05-09T05:00:40.834 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T05:00:40.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T05:00:40.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:40.894 INFO:teuthology.orchestra.run.smithi060.stdout:client4463 2024-05-09T05:00:40.894 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2.client4463/status') 2024-05-09T05:00:40.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:40.954 INFO:teuthology.orchestra.run.smithi060.stdout:instance: client.4463 (3)172.21.15.60:0/794690314 2024-05-09T05:00:40.955 INFO:teuthology.orchestra.run.smithi060.stdout:blocklisted: false 2024-05-09T05:00:40.955 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2024-05-09T05:00:40.955 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2024-05-09T05:00:40.955 INFO:teuthology.orchestra.run.smithi060.stderr:69 bytes copied, 2.4002e-05 s, 2.9 MB/s 2024-05-09T05:00:40.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:40.962 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-05-09T05:00:40.962 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T05:00:40.962 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-05-09T05:00:40.963 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T05:00:41.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.067+0000 7fe2d9cc5640 1 Processor -- start 2024-05-09T05:00:41.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.067+0000 7fe2d9cc5640 1 -- start start 2024-05-09T05:00:41.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.068+0000 7fe2d9cc5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe2d4070150 0x7fe2d4070530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:41.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.068+0000 7fe2d9cc5640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d4097de0 0x7fe2d4071bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:41.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.068+0000 7fe2d9cc5640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2d415c430 con 0x7fe2d4072130 2024-05-09T05:00:41.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.068+0000 7fe2d9cc5640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fe2d409b220 con 0x7fe2d4070150 2024-05-09T05:00:41.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.068+0000 7fe2d9cc5640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe2d4070a70 con 0x7fe2d4097de0 2024-05-09T05:00:41.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.068+0000 7fe2d37fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe2d4070150 0x7fe2d4070530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:41.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.068+0000 7fe2d37fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe2d4070150 0x7fe2d4070530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45956/0 (socket says 172.21.15.60:45956) 2024-05-09T05:00:41.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.068+0000 7fe2d37fe640 1 -- 172.21.15.60:0/2029458376 learned_addr learned my addr 172.21.15.60:0/2029458376 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:41.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.068+0000 7fe2d3fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe2d4072130 0x7fe2d40725e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33212/0 (socket says 172.21.15.60:33212) 2024-05-09T05:00:41.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.068+0000 7fe2d2ffd640 1 --2- 172.21.15.60:0/2029458376 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d4097de0 0x7fe2d4071bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:41.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.068+0000 7fe2d37fe640 1 -- 172.21.15.60:0/2029458376 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d4097de0 msgr2=0x7fe2d4071bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:41.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.068+0000 7fe2d37fe640 1 --2- 172.21.15.60:0/2029458376 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d4097de0 0x7fe2d4071bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:41.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.068+0000 7fe2d37fe640 1 -- 172.21.15.60:0/2029458376 >> v1:172.21.15.60:6789/0 conn(0x7fe2d4072130 legacy=0x7fe2d40725e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:41.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.068+0000 7fe2d27fc640 1 -- 172.21.15.60:0/2029458376 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 373476055 0 0) 0x7fe2d415c430 con 0x7fe2d4072130 2024-05-09T05:00:41.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.068+0000 7fe2d37fe640 1 -- 172.21.15.60:0/2029458376 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d409b470 con 0x7fe2d4070150 2024-05-09T05:00:41.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.069+0000 7fe2d37fe640 1 --2- 172.21.15.60:0/2029458376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2d4070150 0x7fe2d4070530 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400bb10 tx=0x7fe2c402c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b4e12b1d822db1d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:41.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.069+0000 7fe2d27fc640 1 -- 172.21.15.60:0/2029458376 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4036020 con 0x7fe2d4070150 2024-05-09T05:00:41.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.069+0000 7fe2d27fc640 1 -- 172.21.15.60:0/2029458376 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2c4032820 con 0x7fe2d4070150 2024-05-09T05:00:41.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.069+0000 7fe2d27fc640 1 -- 172.21.15.60:0/2029458376 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4032bf0 con 0x7fe2d4070150 2024-05-09T05:00:41.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.070+0000 7fe2d9cc5640 1 -- 172.21.15.60:0/2029458376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2d4070150 msgr2=0x7fe2d4070530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:41.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.070+0000 7fe2d9cc5640 1 --2- 172.21.15.60:0/2029458376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2d4070150 0x7fe2d4070530 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400bb10 tx=0x7fe2c402c920 comp rx=0 tx=0).stop 2024-05-09T05:00:41.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.070+0000 7fe2d9cc5640 1 -- 172.21.15.60:0/2029458376 shutdown_connections 2024-05-09T05:00:41.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.070+0000 7fe2d9cc5640 1 --2- 172.21.15.60:0/2029458376 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d4097de0 0x7fe2d4071bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:41.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.070+0000 7fe2d9cc5640 1 --2- 172.21.15.60:0/2029458376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2d4070150 0x7fe2d4070530 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:41.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.070+0000 7fe2d9cc5640 1 -- 172.21.15.60:0/2029458376 >> 172.21.15.60:0/2029458376 conn(0x7fe2d40972f0 msgr2=0x7fe2d40976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:41.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.070+0000 7fe2d9cc5640 1 -- 172.21.15.60:0/2029458376 shutdown_connections 2024-05-09T05:00:41.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.070+0000 7fe2d9cc5640 1 -- 172.21.15.60:0/2029458376 wait complete. 2024-05-09T05:00:41.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.071+0000 7fe2d9cc5640 1 Processor -- start 2024-05-09T05:00:41.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.071+0000 7fe2d9cc5640 1 -- start start 2024-05-09T05:00:41.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.072+0000 7fe2d9cc5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2d4070150 0x7fe2d40a7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:41.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.072+0000 7fe2d9cc5640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d4072130 0x7fe2d41e9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:41.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.072+0000 7fe2d9cc5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2d4097de0 0x7fe2d41e5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:41.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.072+0000 7fe2d9cc5640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d40b46e0 con 0x7fe2d4097de0 2024-05-09T05:00:41.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.072+0000 7fe2d9cc5640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d406ec30 con 0x7fe2d4070150 2024-05-09T05:00:41.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.072+0000 7fe2d9cc5640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe2d4072cf0 con 0x7fe2d4072130 2024-05-09T05:00:41.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.072+0000 7fe2d3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2d4097de0 0x7fe2d41e5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:41.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.072+0000 7fe2d3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2d4097de0 0x7fe2d41e5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45962/0 (socket says 172.21.15.60:45962) 2024-05-09T05:00:41.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.072+0000 7fe2d3fff640 1 -- 172.21.15.60:0/1647614628 learned_addr learned my addr 172.21.15.60:0/1647614628 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:41.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.072+0000 7fe2d37fe640 1 --2- 172.21.15.60:0/1647614628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2d4070150 0x7fe2d40a7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:41.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.072+0000 7fe2d2ffd640 1 --2- 172.21.15.60:0/1647614628 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d4072130 0x7fe2d41e9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:41.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.072+0000 7fe2d3fff640 1 -- 172.21.15.60:0/1647614628 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d4072130 msgr2=0x7fe2d41e9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:41.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.072+0000 7fe2d3fff640 1 --2- 172.21.15.60:0/1647614628 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d4072130 0x7fe2d41e9240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:41.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.072+0000 7fe2d3fff640 1 -- 172.21.15.60:0/1647614628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2d4070150 msgr2=0x7fe2d40a7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:41.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.072+0000 7fe2d3fff640 1 --2- 172.21.15.60:0/1647614628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2d4070150 0x7fe2d40a7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:41.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.072+0000 7fe2d3fff640 1 -- 172.21.15.60:0/1647614628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d41e9c90 con 0x7fe2d4097de0 2024-05-09T05:00:41.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.072+0000 7fe2d3fff640 1 --2- 172.21.15.60:0/1647614628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2d4097de0 0x7fe2d41e5b10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00c730 tx=0x7fe2cc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:41.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.073+0000 7fe2bbfff640 1 -- 172.21.15.60:0/1647614628 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc018070 con 0x7fe2d4097de0 2024-05-09T05:00:41.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.073+0000 7fe2bbfff640 1 -- 172.21.15.60:0/1647614628 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2cc004d60 con 0x7fe2d4097de0 2024-05-09T05:00:41.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.073+0000 7fe2d9cc5640 1 -- 172.21.15.60:0/1647614628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2d41e9f70 con 0x7fe2d4097de0 2024-05-09T05:00:41.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.073+0000 7fe2bbfff640 1 -- 172.21.15.60:0/1647614628 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc00a880 con 0x7fe2d4097de0 2024-05-09T05:00:41.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.073+0000 7fe2d9cc5640 1 -- 172.21.15.60:0/1647614628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2d41ea810 con 0x7fe2d4097de0 2024-05-09T05:00:41.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.074+0000 7fe2d9cc5640 1 -- 172.21.15.60:0/1647614628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2d4051040 con 0x7fe2d4097de0 2024-05-09T05:00:41.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.074+0000 7fe2bbfff640 1 -- 172.21.15.60:0/1647614628 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe2cc01b070 con 0x7fe2d4097de0 2024-05-09T05:00:41.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.075+0000 7fe2bbfff640 1 --2- 172.21.15.60:0/1647614628 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe2c00762d0 0x7fe2c0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:41.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.075+0000 7fe2bbfff640 1 -- 172.21.15.60:0/1647614628 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe2cc096e10 con 0x7fe2d4097de0 2024-05-09T05:00:41.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.075+0000 7fe2d37fe640 1 --2- 172.21.15.60:0/1647614628 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe2c00762d0 0x7fe2c0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:41.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.076+0000 7fe2d37fe640 1 --2- 172.21.15.60:0/1647614628 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe2c00762d0 0x7fe2c0078790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4037000 tx=0x7fe2c4032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:41.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.079+0000 7fe2bbfff640 1 -- 172.21.15.60:0/1647614628 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe2cc05fde0 con 0x7fe2d4097de0 2024-05-09T05:00:41.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.311+0000 7fe2d9cc5640 1 -- 172.21.15.60:0/1647614628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe2d41ea140 con 0x7fe2d4097de0 2024-05-09T05:00:41.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.311+0000 7fe2bbfff640 1 -- 172.21.15.60:0/1647614628 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7fe2cc0640e0 con 0x7fe2d4097de0 2024-05-09T05:00:41.311 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-09T05:00:41.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.313+0000 7fe2d9cc5640 1 -- 172.21.15.60:0/1647614628 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe2c00762d0 msgr2=0x7fe2c0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:41.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.313+0000 7fe2d9cc5640 1 --2- 172.21.15.60:0/1647614628 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe2c00762d0 0x7fe2c0078790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4037000 tx=0x7fe2c4032000 comp rx=0 tx=0).stop 2024-05-09T05:00:41.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.313+0000 7fe2d9cc5640 1 -- 172.21.15.60:0/1647614628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2d4097de0 msgr2=0x7fe2d41e5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:41.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.313+0000 7fe2d9cc5640 1 --2- 172.21.15.60:0/1647614628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2d4097de0 0x7fe2d41e5b10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00c730 tx=0x7fe2cc00cbf0 comp rx=0 tx=0).stop 2024-05-09T05:00:41.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.313+0000 7fe2d9cc5640 1 -- 172.21.15.60:0/1647614628 shutdown_connections 2024-05-09T05:00:41.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.313+0000 7fe2d9cc5640 1 --2- 172.21.15.60:0/1647614628 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe2c00762d0 0x7fe2c0078790 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:41.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.313+0000 7fe2d9cc5640 1 --2- 172.21.15.60:0/1647614628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe2d4097de0 0x7fe2d41e5b10 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:41.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.313+0000 7fe2d9cc5640 1 --2- 172.21.15.60:0/1647614628 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe2d4072130 0x7fe2d41e9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:41.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.313+0000 7fe2d9cc5640 1 --2- 172.21.15.60:0/1647614628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2d4070150 0x7fe2d40a7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:41.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.313+0000 7fe2d9cc5640 1 -- 172.21.15.60:0/1647614628 >> 172.21.15.60:0/1647614628 conn(0x7fe2d40972f0 msgr2=0x7fe2d415a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:41.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.313+0000 7fe2d9cc5640 1 -- 172.21.15.60:0/1647614628 shutdown_connections 2024-05-09T05:00:41.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41.313+0000 7fe2d9cc5640 1 -- 172.21.15.60:0/1647614628 wait complete. 2024-05-09T05:00:41.324 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T05:00:41.324 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T05:00:41.324 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2024-05-09T05:00:41.324 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-05-09T05:00:41.325 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-05-09T05:00:41.325 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T05:00:41.325 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T05:00:41.325 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T05:00:41.325 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-05-09T05:00:41.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:41.326 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2024-05-09T05:00:41.343 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T05:00:41.343 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T05:00:41.343 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T05:00:41.343 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:41.343 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2024-05-09T05:00:41.343 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:41.344 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:00:41.344 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:41.344 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2024-05-09T05:00:41.344 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:00:41.344 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:41.344 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2024-05-09T05:00:41.344 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T05:00:41.344 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:41.344 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2024-05-09T05:00:41.344 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T05:00:41.344 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 42722sec preferred_lft 42722sec 2024-05-09T05:00:41.344 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2024-05-09T05:00:41.344 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:41.344 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:00:41.345 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:41.345 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2024-05-09T05:00:41.345 INFO:teuthology.orchestra.run.smithi060.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T05:00:41.345 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 42:54:29:48:e0:84 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:41.345 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T05:00:41.345 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:41.345 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::4889:3fff:fe4f:b9fb/64 scope link 2024-05-09T05:00:41.345 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:41.345 INFO:teuthology.orchestra.run.smithi060.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T05:00:41.345 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 42:54:29:48:e0:84 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T05:00:41.345 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::4054:29ff:fe48:e084/64 scope link tentative 2024-05-09T05:00:41.345 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:41.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:41.346 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T05:00:41.407 INFO:teuthology.orchestra.run.smithi060.stdout:1 2024-05-09T05:00:41.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:41.409 DEBUG:teuthology.orchestra.run.smithi060:> route 2024-05-09T05:00:41.467 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2024-05-09T05:00:41.467 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T05:00:41.467 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T05:00:41.467 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T05:00:41.467 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T05:00:41.467 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:41.467 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-05-09T05:00:41.468 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T05:00:41.468 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T05:00:41.468 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T05:00:41.468 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-05-09T05:00:41.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:41.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:41.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:41.598 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2024-05-09T05:00:41.614 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T05:00:41.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:41.614 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list-id 2024-05-09T05:00:41.670 INFO:teuthology.orchestra.run.smithi060.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-05-09T05:00:41.671 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:41.671 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-05-09T05:00:41.671 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T05:00:41.671 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-05-09T05:00:41.671 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-05-09T05:00:41.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:41.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:41.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:41.792 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-05-09T05:00:41.817 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T05:00:41.817 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T05:00:41.817 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T05:00:41.817 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:41.817 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2024-05-09T05:00:41.817 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:41.817 INFO:teuthology.orchestra.run.smithi060.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T05:00:41.817 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether ce:7c:6c:fd:3d:c6 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-09T05:00:41.817 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-09T05:00:41.817 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:41.818 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::cc7c:6cff:fefd:3dc6/64 scope link tentative 2024-05-09T05:00:41.818 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:41.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:41.819 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-05-09T05:00:41.884 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T05:00:41.884 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T05:00:41.885 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T05:00:41.885 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:41.885 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2024-05-09T05:00:41.885 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:41.885 INFO:teuthology.orchestra.run.smithi060.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T05:00:41.885 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether ce:7c:6c:fd:3d:c6 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-09T05:00:41.885 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-09T05:00:41.885 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:41.885 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::cc7c:6cff:fefd:3dc6/64 scope link tentative 2024-05-09T05:00:41.885 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:41.886 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-05-09T05:00:41.887 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:41.887 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-05-09T05:00:41.887 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-05-09T05:00:41.887 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-05-09T05:00:41.887 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-05-09T05:00:41.887 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-05-09T05:00:41.887 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-05-09T05:00:41.887 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-05-09T05:00:41.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:42.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:42.043 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:42.043 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-05-09T05:00:42.043 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.1 up 2024-05-09T05:00:42.043 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set dev brx.1 master ceph-brx 2024-05-09T05:00:42.043 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-05-09T05:00:42.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:42.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:42.123 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T05:00:42.123 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-05-09T05:00:42.150 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-05-09T05:00:42.151 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T05:00:42.151 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-05-09T05:00:42.206 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-05-09T05:00:42.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:42.206 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T05:00:42.291 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T05:00:42.291 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T05:00:42.298 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T05:00:42.298 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T05:00:42.298 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.114:3300,172.21.15.114:3301" 2024-05-09T05:00:42.298 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 2@0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2.cephfs=/ 2024-05-09T05:00:42.298 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.60:3300/172.21.15.114:3300/172.21.15.114:3301" will pass to kernel 2024-05-09T05:00:42.301 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-05-09T05:00:42.301 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2024-05-09T05:00:42.301 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-09T05:00:42.301 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2024-05-09T05:00:42.301 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-09T05:00:42.301 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-09T05:00:42.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:42.301 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-05-09T05:00:42.333 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T05:00:42.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T05:00:42.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:42.431 INFO:teuthology.orchestra.run.smithi060.stdout:client4695 2024-05-09T05:00:42.431 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:00:42.432 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:00:42.448 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T05:00:42.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T05:00:42.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:42.545 INFO:teuthology.orchestra.run.smithi060.stdout:client4695 2024-05-09T05:00:42.546 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T05:00:42.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T05:00:42.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:42.603 INFO:teuthology.orchestra.run.smithi060.stdout:client4695 2024-05-09T05:00:42.604 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2.client4695/status') 2024-05-09T05:00:42.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:42.664 INFO:teuthology.orchestra.run.smithi060.stdout:instance: client.4695 (3)192.168.0.2:0/2344299263 2024-05-09T05:00:42.664 INFO:teuthology.orchestra.run.smithi060.stdout:blocklisted: false 2024-05-09T05:00:42.664 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2024-05-09T05:00:42.664 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2024-05-09T05:00:42.665 INFO:teuthology.orchestra.run.smithi060.stderr:69 bytes copied, 2.4707e-05 s, 2.8 MB/s 2024-05-09T05:00:42.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:42.672 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-05-09T05:00:42.672 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T05:00:42.672 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-05-09T05:00:42.672 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T05:00:42.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.776+0000 7f54a0b22640 1 Processor -- start 2024-05-09T05:00:42.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.776+0000 7f54a0b22640 1 -- start start 2024-05-09T05:00:42.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f54a0b22640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f549c0b4d40 0x7f549c09e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:42.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f54a0b22640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f549c071f30 con 0x7f549c09c8b0 2024-05-09T05:00:42.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f54a0b22640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f549c072210 con 0x7f549c15f3e0 2024-05-09T05:00:42.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f54a0b22640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f549c071ce0 con 0x7f549c0b4d40 2024-05-09T05:00:42.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f549ad9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f549c09c8b0 0x7f549c09e640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33222/0 (socket says 172.21.15.60:33222) 2024-05-09T05:00:42.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f549ad9d640 1 -- 172.21.15.60:0/2818177103 learned_addr learned my addr 172.21.15.60:0/2818177103 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:42.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f5499d9b640 1 --2- 172.21.15.60:0/2818177103 >> v2:172.21.15.114:3300/0 conn(0x7f549c0b4d40 0x7f549c09e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:42.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f549959a640 1 -- 172.21.15.60:0/2818177103 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2465093848 0 0) 0x7f549c071f30 con 0x7f549c09c8b0 2024-05-09T05:00:42.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f549959a640 1 -- 172.21.15.60:0/2818177103 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5484003520 con 0x7f549c09c8b0 2024-05-09T05:00:42.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f549959a640 1 -- 172.21.15.60:0/2818177103 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 822472750 0 0) 0x7f5484003520 con 0x7f549c09c8b0 2024-05-09T05:00:42.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f549959a640 1 -- 172.21.15.60:0/2818177103 >> v2:172.21.15.114:3300/0 conn(0x7f549c0b4d40 msgr2=0x7f549c09e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:42.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f549959a640 1 --2- 172.21.15.60:0/2818177103 >> v2:172.21.15.114:3300/0 conn(0x7f549c0b4d40 0x7f549c09e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:42.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f549959a640 1 -- 172.21.15.60:0/2818177103 >> v1:172.21.15.114:6789/0 conn(0x7f549c15f3e0 legacy=0x7f549c0b4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:42.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f549959a640 1 -- 172.21.15.60:0/2818177103 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f549c0724f0 con 0x7f549c09c8b0 2024-05-09T05:00:42.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f549959a640 1 -- 172.21.15.60:0/2818177103 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f5494001a70 con 0x7f549c09c8b0 2024-05-09T05:00:42.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f549959a640 1 -- 172.21.15.60:0/2818177103 >> v1:172.21.15.60:6789/0 conn(0x7f549c09c8b0 legacy=0x7f549c09e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:42.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f549959a640 1 --2- 172.21.15.60:0/2818177103 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5484003a50 0x7f5484003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:42.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f549959a640 1 --2- 172.21.15.60:0/2818177103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5484004800 0x7f5484004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:42.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f549959a640 1 --2- 172.21.15.60:0/2818177103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5484005210 0x7f548400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:42.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f549959a640 1 -- 172.21.15.60:0/2818177103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f549c071ce0 con 0x7f5484004800 2024-05-09T05:00:42.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f549959a640 1 -- 172.21.15.60:0/2818177103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f548400c140 con 0x7f5484005210 2024-05-09T05:00:42.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f549959a640 1 -- 172.21.15.60:0/2818177103 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f548400c340 con 0x7f5484003a50 2024-05-09T05:00:42.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.777+0000 7f5499d9b640 1 --2- 172.21.15.60:0/2818177103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5484004800 0x7f5484004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:42.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.778+0000 7f549a59c640 1 --2- 172.21.15.60:0/2818177103 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5484003a50 0x7f5484003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:42.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.778+0000 7f549ad9d640 1 --2- 172.21.15.60:0/2818177103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5484005210 0x7f548400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:42.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.778+0000 7f5499d9b640 1 -- 172.21.15.60:0/2818177103 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5484003a50 msgr2=0x7f5484003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:42.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.778+0000 7f5499d9b640 1 --2- 172.21.15.60:0/2818177103 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5484003a50 0x7f5484003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:42.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.778+0000 7f5499d9b640 1 -- 172.21.15.60:0/2818177103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5484005210 msgr2=0x7f548400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:42.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.778+0000 7f5499d9b640 1 --2- 172.21.15.60:0/2818177103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5484005210 0x7f548400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:42.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.778+0000 7f5499d9b640 1 -- 172.21.15.60:0/2818177103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5484003520 con 0x7f5484004800 2024-05-09T05:00:42.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.778+0000 7f549ad9d640 1 -- 172.21.15.60:0/2818177103 reap_dead start 2024-05-09T05:00:42.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.778+0000 7f5499d9b640 1 --2- 172.21.15.60:0/2818177103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5484004800 0x7f5484004c80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5490009ac0 tx=0x7f5484005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef68583f9955c890 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:42.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.779+0000 7f549959a640 1 -- 172.21.15.60:0/2818177103 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5490036070 con 0x7f5484004800 2024-05-09T05:00:42.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.779+0000 7f549959a640 1 -- 172.21.15.60:0/2818177103 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5490031400 con 0x7f5484004800 2024-05-09T05:00:42.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.779+0000 7f549959a640 1 -- 172.21.15.60:0/2818177103 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5490009690 con 0x7f5484004800 2024-05-09T05:00:42.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.779+0000 7f54a0b22640 1 -- 172.21.15.60:0/2818177103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5484004800 msgr2=0x7f5484004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:42.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.779+0000 7f54a0b22640 1 --2- 172.21.15.60:0/2818177103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5484004800 0x7f5484004c80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5490009ac0 tx=0x7f5484005830 comp rx=0 tx=0).stop 2024-05-09T05:00:42.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.779+0000 7f54a0b22640 1 -- 172.21.15.60:0/2818177103 shutdown_connections 2024-05-09T05:00:42.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.779+0000 7f54a0b22640 1 --2- 172.21.15.60:0/2818177103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5484004800 0x7f5484004c80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:42.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.779+0000 7f54a0b22640 1 -- 172.21.15.60:0/2818177103 >> 172.21.15.60:0/2818177103 conn(0x7f549c0d1470 msgr2=0x7f549c0d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:42.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.779+0000 7f54a0b22640 1 -- 172.21.15.60:0/2818177103 shutdown_connections 2024-05-09T05:00:42.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.779+0000 7f54a0b22640 1 -- 172.21.15.60:0/2818177103 wait complete. 2024-05-09T05:00:42.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.781+0000 7f54a0b22640 1 Processor -- start 2024-05-09T05:00:42.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.781+0000 7f54a0b22640 1 -- start start 2024-05-09T05:00:42.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.781+0000 7f54a0b22640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5484004800 0x7f549c0ca960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:42.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.781+0000 7f54a0b22640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f549c0a9330 0x7f549c0c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:42.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.781+0000 7f54a0b22640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549c0c9340 0x7f549c0cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:42.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.781+0000 7f54a0b22640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f549c15f7c0 con 0x7f549c0c9340 2024-05-09T05:00:42.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.781+0000 7f54a0b22640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f549c072ab0 con 0x7f5484004800 2024-05-09T05:00:42.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.781+0000 7f54a0b22640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f549c072fd0 con 0x7f549c0a9330 2024-05-09T05:00:42.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.781+0000 7f549ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549c0c9340 0x7f549c0cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:42.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.781+0000 7f549ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549c0c9340 0x7f549c0cb610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45966/0 (socket says 172.21.15.60:45966) 2024-05-09T05:00:42.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.781+0000 7f549ad9d640 1 -- 172.21.15.60:0/3539406050 learned_addr learned my addr 172.21.15.60:0/3539406050 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:42.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.782+0000 7f549a59c640 1 --2- 172.21.15.60:0/3539406050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5484004800 0x7f549c0ca960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:42.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.782+0000 7f5499d9b640 1 --2- 172.21.15.60:0/3539406050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f549c0a9330 0x7f549c0c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:42.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.782+0000 7f549ad9d640 1 -- 172.21.15.60:0/3539406050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f549c0a9330 msgr2=0x7f549c0c8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:42.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.782+0000 7f549ad9d640 1 --2- 172.21.15.60:0/3539406050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f549c0a9330 0x7f549c0c8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:42.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.782+0000 7f549ad9d640 1 -- 172.21.15.60:0/3539406050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5484004800 msgr2=0x7f549c0ca960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:42.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.782+0000 7f549ad9d640 1 --2- 172.21.15.60:0/3539406050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5484004800 0x7f549c0ca960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:42.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.782+0000 7f549ad9d640 1 -- 172.21.15.60:0/3539406050 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f549c0c98c0 con 0x7f549c0c9340 2024-05-09T05:00:42.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.782+0000 7f5499d9b640 1 --2- 172.21.15.60:0/3539406050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f549c0a9330 0x7f549c0c8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:42.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.782+0000 7f549ad9d640 1 --2- 172.21.15.60:0/3539406050 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549c0c9340 0x7f549c0cb610 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f549c09d960 tx=0x7f5494007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:42.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.782+0000 7f548affd640 1 -- 172.21.15.60:0/3539406050 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549400aa30 con 0x7f549c0c9340 2024-05-09T05:00:42.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.782+0000 7f548affd640 1 -- 172.21.15.60:0/3539406050 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5494019070 con 0x7f549c0c9340 2024-05-09T05:00:42.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.782+0000 7f548affd640 1 -- 172.21.15.60:0/3539406050 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549400ae30 con 0x7f549c0c9340 2024-05-09T05:00:42.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.782+0000 7f54a0b22640 1 -- 172.21.15.60:0/3539406050 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f549c0c9ba0 con 0x7f549c0c9340 2024-05-09T05:00:42.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.782+0000 7f54a0b22640 1 -- 172.21.15.60:0/3539406050 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f549c0ca160 con 0x7f549c0c9340 2024-05-09T05:00:42.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.783+0000 7f54a0b22640 1 -- 172.21.15.60:0/3539406050 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f549c0ca470 con 0x7f549c0c9340 2024-05-09T05:00:42.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.784+0000 7f548affd640 1 -- 172.21.15.60:0/3539406050 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f549401e460 con 0x7f549c0c9340 2024-05-09T05:00:42.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.784+0000 7f548affd640 1 --2- 172.21.15.60:0/3539406050 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f547c076300 0x7f547c0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:42.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.784+0000 7f548affd640 1 -- 172.21.15.60:0/3539406050 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5494097cd0 con 0x7f549c0c9340 2024-05-09T05:00:42.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.784+0000 7f549a59c640 1 --2- 172.21.15.60:0/3539406050 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f547c076300 0x7f547c0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:42.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.785+0000 7f549a59c640 1 --2- 172.21.15.60:0/3539406050 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f547c076300 0x7f547c0787c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f548c000c00 tx=0x7f548c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:42.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:42.789+0000 7f548affd640 1 -- 172.21.15.60:0/3539406050 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f54940612f0 con 0x7f549c0c9340 2024-05-09T05:00:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:43.019+0000 7f54a0b22640 1 -- 172.21.15.60:0/3539406050 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f549c051040 con 0x7f549c0c9340 2024-05-09T05:00:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:43.019+0000 7f548affd640 1 -- 172.21.15.60:0/3539406050 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f5494060bf0 con 0x7f549c0c9340 2024-05-09T05:00:43.019 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-09T05:00:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:43.021+0000 7f54a0b22640 1 -- 172.21.15.60:0/3539406050 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f547c076300 msgr2=0x7f547c0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:43.021+0000 7f54a0b22640 1 --2- 172.21.15.60:0/3539406050 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f547c076300 0x7f547c0787c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f548c000c00 tx=0x7f548c009040 comp rx=0 tx=0).stop 2024-05-09T05:00:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:43.021+0000 7f54a0b22640 1 -- 172.21.15.60:0/3539406050 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549c0c9340 msgr2=0x7f549c0cb610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:43.021+0000 7f54a0b22640 1 --2- 172.21.15.60:0/3539406050 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549c0c9340 0x7f549c0cb610 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f549c09d960 tx=0x7f5494007a90 comp rx=0 tx=0).stop 2024-05-09T05:00:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:43.021+0000 7f54a0b22640 1 -- 172.21.15.60:0/3539406050 shutdown_connections 2024-05-09T05:00:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:43.021+0000 7f54a0b22640 1 --2- 172.21.15.60:0/3539406050 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f547c076300 0x7f547c0787c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:43.021+0000 7f54a0b22640 1 --2- 172.21.15.60:0/3539406050 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549c0c9340 0x7f549c0cb610 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:43.021+0000 7f54a0b22640 1 --2- 172.21.15.60:0/3539406050 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f549c0a9330 0x7f549c0c8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:43.021+0000 7f54a0b22640 1 --2- 172.21.15.60:0/3539406050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5484004800 0x7f549c0ca960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:43.021+0000 7f54a0b22640 1 -- 172.21.15.60:0/3539406050 >> 172.21.15.60:0/3539406050 conn(0x7f549c0d1470 msgr2=0x7f549c0b39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:43.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:43.021+0000 7f54a0b22640 1 -- 172.21.15.60:0/3539406050 shutdown_connections 2024-05-09T05:00:43.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:43.021+0000 7f54a0b22640 1 -- 172.21.15.60:0/3539406050 wait complete. 2024-05-09T05:00:43.031 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T05:00:43.031 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T05:00:43.031 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-05-09T05:00:43.031 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-05-09T05:00:43.032 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-05-09T05:00:43.032 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T05:00:43.032 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T05:00:43.032 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T05:00:43.032 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-05-09T05:00:43.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:43.032 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-05-09T05:00:43.052 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T05:00:43.052 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T05:00:43.052 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T05:00:43.052 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:43.052 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-05-09T05:00:43.052 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:43.053 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:00:43.053 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:43.053 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-05-09T05:00:43.053 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:00:43.053 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:43.053 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-05-09T05:00:43.053 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:00:43.053 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:43.053 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T05:00:43.053 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:43.053 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-09T05:00:43.053 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42751sec preferred_lft 42751sec 2024-05-09T05:00:43.053 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link 2024-05-09T05:00:43.053 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:43.054 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-09T05:00:43.054 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:43.054 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T05:00:43.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add name ceph-brx type bridge 2024-05-09T05:00:43.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr flush dev ceph-brx 2024-05-09T05:00:43.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set ceph-brx up 2024-05-09T05:00:43.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-09T05:00:43.054 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T05:00:43.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:43.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:43.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:43.350 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T05:00:43.376 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-05-09T05:00:43.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:43.377 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-05-09T05:00:43.443 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-05-09T05:00:43.443 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T05:00:43.443 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T05:00:43.443 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T05:00:43.443 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T05:00:43.443 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:43.444 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T05:00:43.444 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T05:00:43.444 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T05:00:43.444 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T05:00:43.444 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T05:00:43.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:43.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:43.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:43.746 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-09T05:00:43.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:43.761 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list-id 2024-05-09T05:00:43.817 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:43.817 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T05:00:43.817 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T05:00:43.817 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-05-09T05:00:43.817 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T05:00:43.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:43.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:43.934 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-05-09T05:00:43.934 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:43.934 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T05:00:43.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-05-09T05:00:43.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-09T05:00:43.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-05-09T05:00:43.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-05-09T05:00:43.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-05-09T05:00:43.935 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T05:00:43.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:44.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:44.060 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:44.060 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T05:00:44.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-05-09T05:00:44.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set dev brx.0 master ceph-brx 2024-05-09T05:00:44.060 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T05:00:44.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:44.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:44.137 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T05:00:44.137 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-05-09T05:00:44.161 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-05-09T05:00:44.162 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T05:00:44.162 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-05-09T05:00:44.217 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-05-09T05:00:44.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:44.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T05:00:44.310 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T05:00:44.310 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T05:00:44.414 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T05:00:44.414 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T05:00:44.414 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.114:3300,172.21.15.114:3301" 2024-05-09T05:00:44.414 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with new device syntax: 1@0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2.cephfs=/ 2024-05-09T05:00:44.414 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.60:3300/172.21.15.114:3300/172.21.15.114:3301" will pass to kernel 2024-05-09T05:00:44.415 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-05-09T05:00:44.415 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-05-09T05:00:44.415 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-09T05:00:44.415 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-05-09T05:00:44.415 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-09T05:00:44.417 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-09T05:00:44.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:44.417 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-05-09T05:00:44.441 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T05:00:44.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:44 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T05:00:44.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:44.538 INFO:teuthology.orchestra.run.smithi114.stdout:client4704 2024-05-09T05:00:44.538 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T05:00:44.538 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:00:44.554 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T05:00:44.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:44 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T05:00:44.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:44.651 INFO:teuthology.orchestra.run.smithi114.stdout:client4704 2024-05-09T05:00:44.651 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T05:00:44.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:44 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T05:00:44.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:44.708 INFO:teuthology.orchestra.run.smithi114.stdout:client4704 2024-05-09T05:00:44.708 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2.client4704/status') 2024-05-09T05:00:44.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:44.767 INFO:teuthology.orchestra.run.smithi114.stdout:instance: client.4704 (3)172.21.15.114:0/2210310443 2024-05-09T05:00:44.767 INFO:teuthology.orchestra.run.smithi114.stdout:blocklisted: false 2024-05-09T05:00:44.767 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records in 2024-05-09T05:00:44.768 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records out 2024-05-09T05:00:44.768 INFO:teuthology.orchestra.run.smithi114.stderr:71 bytes copied, 2.4504e-05 s, 2.9 MB/s 2024-05-09T05:00:44.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:44.775 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-05-09T05:00:44.775 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T05:00:44.775 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-05-09T05:00:44.775 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T05:00:44.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.895+0000 7f1ae6c59640 1 Processor -- start 2024-05-09T05:00:44.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.895+0000 7f1ae6c59640 1 -- start start 2024-05-09T05:00:44.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.896+0000 7f1ae6c59640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1ae01606f0 0x7f1ae00d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:44.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.896+0000 7f1ae6c59640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1ae006f9f0 0x7f1ae00d3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:44.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.896+0000 7f1ae6c59640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ae006ff60 con 0x7f1ae015f300 2024-05-09T05:00:44.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.896+0000 7f1ae6c59640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f1ae0070240 con 0x7f1ae006f9f0 2024-05-09T05:00:44.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.896+0000 7f1ae6c59640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f1ae0070520 con 0x7f1ae01606f0 2024-05-09T05:00:44.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.896+0000 7f1ae51f6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1ae006f9f0 0x7f1ae00d3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:44.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.896+0000 7f1ae49f5640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1ae015f300 0x7f1ae01639f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33238/0 (socket says 172.21.15.60:33238) 2024-05-09T05:00:44.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.896+0000 7f1ae49f5640 1 -- 172.21.15.60:0/4156302522 learned_addr learned my addr 172.21.15.60:0/4156302522 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:44.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.896+0000 7f1adffff640 1 --2- 172.21.15.60:0/4156302522 >> v2:172.21.15.114:3300/0 conn(0x7f1ae01606f0 0x7f1ae00d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:44.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.896+0000 7f1ae51f6640 1 -- 172.21.15.60:0/4156302522 >> v2:172.21.15.114:3300/0 conn(0x7f1ae01606f0 msgr2=0x7f1ae00d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:44.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.896+0000 7f1ae51f6640 1 --2- 172.21.15.60:0/4156302522 >> v2:172.21.15.114:3300/0 conn(0x7f1ae01606f0 0x7f1ae00d33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:44.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.896+0000 7f1ae51f6640 1 -- 172.21.15.60:0/4156302522 >> v1:172.21.15.60:6789/0 conn(0x7f1ae015f300 legacy=0x7f1ae01639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:44.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.896+0000 7f1ae51f6640 1 -- 172.21.15.60:0/4156302522 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae006e740 con 0x7f1ae006f9f0 2024-05-09T05:00:44.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.897+0000 7f1ae51f6640 1 --2- 172.21.15.60:0/4156302522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae006f9f0 0x7f1ae00d3930 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1ad80097a0 tx=0x7f1ad802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3488712f019902d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:44.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.897+0000 7f1adf7fe640 1 -- 172.21.15.60:0/4156302522 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8037070 con 0x7f1ae006f9f0 2024-05-09T05:00:44.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.897+0000 7f1adf7fe640 1 -- 172.21.15.60:0/4156302522 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ad8032440 con 0x7f1ae006f9f0 2024-05-09T05:00:44.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.897+0000 7f1adf7fe640 1 -- 172.21.15.60:0/4156302522 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8032810 con 0x7f1ae006f9f0 2024-05-09T05:00:44.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.898+0000 7f1ae6c59640 1 -- 172.21.15.60:0/4156302522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae006f9f0 msgr2=0x7f1ae00d3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:44.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.898+0000 7f1ae6c59640 1 --2- 172.21.15.60:0/4156302522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae006f9f0 0x7f1ae00d3930 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1ad80097a0 tx=0x7f1ad802a1e0 comp rx=0 tx=0).stop 2024-05-09T05:00:44.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.898+0000 7f1ae6c59640 1 -- 172.21.15.60:0/4156302522 shutdown_connections 2024-05-09T05:00:44.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.898+0000 7f1ae6c59640 1 --2- 172.21.15.60:0/4156302522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae006f9f0 0x7f1ae00d3930 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:44.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.898+0000 7f1ae6c59640 1 --2- 172.21.15.60:0/4156302522 >> v2:172.21.15.114:3300/0 conn(0x7f1ae01606f0 0x7f1ae00d33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:44.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.898+0000 7f1ae6c59640 1 -- 172.21.15.60:0/4156302522 >> 172.21.15.60:0/4156302522 conn(0x7f1ae009b850 msgr2=0x7f1ae009bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:44.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.898+0000 7f1ae6c59640 1 -- 172.21.15.60:0/4156302522 shutdown_connections 2024-05-09T05:00:44.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.898+0000 7f1ae6c59640 1 -- 172.21.15.60:0/4156302522 wait complete. 2024-05-09T05:00:44.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.899+0000 7f1ae6c59640 1 Processor -- start 2024-05-09T05:00:44.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.899+0000 7f1ae6c59640 1 -- start start 2024-05-09T05:00:44.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.900+0000 7f1ae6c59640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ae006f9f0 0x7f1ae00b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:44.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.900+0000 7f1ae6c59640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae015f300 0x7f1ae015c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:44.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.900+0000 7f1ae6c59640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ae01606f0 0x7f1ae0160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:44.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.900+0000 7f1ae6c59640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae006f0c0 con 0x7f1ae015f300 2024-05-09T05:00:44.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.900+0000 7f1ae6c59640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae006ef40 con 0x7f1ae01606f0 2024-05-09T05:00:44.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.900+0000 7f1ae6c59640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1ae006f240 con 0x7f1ae006f9f0 2024-05-09T05:00:44.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.900+0000 7f1adffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae015f300 0x7f1ae015c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:44.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.900+0000 7f1adffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae015f300 0x7f1ae015c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45988/0 (socket says 172.21.15.60:45988) 2024-05-09T05:00:44.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.900+0000 7f1adffff640 1 -- 172.21.15.60:0/1644545175 learned_addr learned my addr 172.21.15.60:0/1644545175 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:44.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.900+0000 7f1ae49f5640 1 --2- 172.21.15.60:0/1644545175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ae006f9f0 0x7f1ae00b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:44.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.900+0000 7f1ae51f6640 1 --2- 172.21.15.60:0/1644545175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ae01606f0 0x7f1ae0160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:44.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.900+0000 7f1adffff640 1 -- 172.21.15.60:0/1644545175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ae006f9f0 msgr2=0x7f1ae00b1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:44.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.900+0000 7f1adffff640 1 --2- 172.21.15.60:0/1644545175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ae006f9f0 0x7f1ae00b1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:44.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.900+0000 7f1adffff640 1 -- 172.21.15.60:0/1644545175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ae01606f0 msgr2=0x7f1ae0160ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:44.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.900+0000 7f1adffff640 1 --2- 172.21.15.60:0/1644545175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ae01606f0 0x7f1ae0160ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:44.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.900+0000 7f1adffff640 1 -- 172.21.15.60:0/1644545175 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae0161490 con 0x7f1ae015f300 2024-05-09T05:00:44.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.900+0000 7f1ae51f6640 1 --2- 172.21.15.60:0/1644545175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ae01606f0 0x7f1ae0160ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:44.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.900+0000 7f1ae49f5640 1 --2- 172.21.15.60:0/1644545175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ae006f9f0 0x7f1ae00b1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:44.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.900+0000 7f1adffff640 1 --2- 172.21.15.60:0/1644545175 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae015f300 0x7f1ae015c320 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000efc0 tx=0x7f1ad000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:44.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.901+0000 7f1add7fa640 1 -- 172.21.15.60:0/1644545175 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0018020 con 0x7f1ae015f300 2024-05-09T05:00:44.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.901+0000 7f1add7fa640 1 -- 172.21.15.60:0/1644545175 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ad0014480 con 0x7f1ae015f300 2024-05-09T05:00:44.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.901+0000 7f1ae6c59640 1 -- 172.21.15.60:0/1644545175 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ae0161770 con 0x7f1ae015f300 2024-05-09T05:00:44.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.901+0000 7f1add7fa640 1 -- 172.21.15.60:0/1644545175 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0014850 con 0x7f1ae015f300 2024-05-09T05:00:44.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.901+0000 7f1ae6c59640 1 -- 172.21.15.60:0/1644545175 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ae00af420 con 0x7f1ae015f300 2024-05-09T05:00:44.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.902+0000 7f1ae6c59640 1 -- 172.21.15.60:0/1644545175 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ae0051040 con 0x7f1ae015f300 2024-05-09T05:00:44.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.902+0000 7f1add7fa640 1 -- 172.21.15.60:0/1644545175 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1ad0005d10 con 0x7f1ae015f300 2024-05-09T05:00:44.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.903+0000 7f1add7fa640 1 --2- 172.21.15.60:0/1644545175 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1ac00762d0 0x7f1ac0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:44.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.903+0000 7f1add7fa640 1 -- 172.21.15.60:0/1644545175 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f1ad0098710 con 0x7f1ae015f300 2024-05-09T05:00:44.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.903+0000 7f1ae49f5640 1 --2- 172.21.15.60:0/1644545175 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1ac00762d0 0x7f1ac0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:44.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.904+0000 7f1ae49f5640 1 --2- 172.21.15.60:0/1644545175 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1ac00762d0 0x7f1ac0078790 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8000f60 tx=0x7f1ac8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:44.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:44.907+0000 7f1add7fa640 1 -- 172.21.15.60:0/1644545175 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1ad0022030 con 0x7f1ae015f300 2024-05-09T05:00:45.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:45.139+0000 7f1ae6c59640 1 -- 172.21.15.60:0/1644545175 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1ae009f8e0 con 0x7f1ae015f300 2024-05-09T05:00:45.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:45.139+0000 7f1add7fa640 1 -- 172.21.15.60:0/1644545175 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f1ad0010040 con 0x7f1ae015f300 2024-05-09T05:00:45.139 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-09T05:00:45.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:45.141+0000 7f1ae6c59640 1 -- 172.21.15.60:0/1644545175 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1ac00762d0 msgr2=0x7f1ac0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:45.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:45.141+0000 7f1ae6c59640 1 --2- 172.21.15.60:0/1644545175 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1ac00762d0 0x7f1ac0078790 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8000f60 tx=0x7f1ac8009040 comp rx=0 tx=0).stop 2024-05-09T05:00:45.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:45.141+0000 7f1ae6c59640 1 -- 172.21.15.60:0/1644545175 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae015f300 msgr2=0x7f1ae015c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:45.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:45.141+0000 7f1ae6c59640 1 --2- 172.21.15.60:0/1644545175 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae015f300 0x7f1ae015c320 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000efc0 tx=0x7f1ad000c490 comp rx=0 tx=0).stop 2024-05-09T05:00:45.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:45.141+0000 7f1ae6c59640 1 -- 172.21.15.60:0/1644545175 shutdown_connections 2024-05-09T05:00:45.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:45.141+0000 7f1ae6c59640 1 --2- 172.21.15.60:0/1644545175 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1ac00762d0 0x7f1ac0078790 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:45.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:45.141+0000 7f1ae6c59640 1 --2- 172.21.15.60:0/1644545175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ae01606f0 0x7f1ae0160ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:45.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:45.141+0000 7f1ae6c59640 1 --2- 172.21.15.60:0/1644545175 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae015f300 0x7f1ae015c320 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:45.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:45.141+0000 7f1ae6c59640 1 --2- 172.21.15.60:0/1644545175 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ae006f9f0 0x7f1ae00b1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:45.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:45.141+0000 7f1ae6c59640 1 -- 172.21.15.60:0/1644545175 >> 172.21.15.60:0/1644545175 conn(0x7f1ae009b850 msgr2=0x7f1ae00d3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:45.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:45.141+0000 7f1ae6c59640 1 -- 172.21.15.60:0/1644545175 shutdown_connections 2024-05-09T05:00:45.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:45.141+0000 7f1ae6c59640 1 -- 172.21.15.60:0/1644545175 wait complete. 2024-05-09T05:00:45.151 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T05:00:45.152 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T05:00:45.152 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-05-09T05:00:45.152 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-05-09T05:00:45.152 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-05-09T05:00:45.152 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T05:00:45.152 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T05:00:45.152 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T05:00:45.152 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-05-09T05:00:45.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:45.153 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-05-09T05:00:45.169 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T05:00:45.169 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T05:00:45.169 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T05:00:45.169 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:45.169 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-05-09T05:00:45.169 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:45.170 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:00:45.170 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:45.170 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-05-09T05:00:45.170 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:00:45.170 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:45.170 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-05-09T05:00:45.170 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:00:45.170 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:45.170 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T05:00:45.170 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:45.170 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-09T05:00:45.171 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42749sec preferred_lft 42749sec 2024-05-09T05:00:45.171 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link 2024-05-09T05:00:45.171 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:45.171 INFO:teuthology.orchestra.run.smithi114.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T05:00:45.171 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 86:fd:2d:94:26:d8 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:00:45.171 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T05:00:45.171 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:45.171 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::c483:94ff:fe0e:fa21/64 scope link 2024-05-09T05:00:45.171 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:45.171 INFO:teuthology.orchestra.run.smithi114.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T05:00:45.171 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 86:fd:2d:94:26:d8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T05:00:45.171 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::84fd:2dff:fe94:26d8/64 scope link tentative 2024-05-09T05:00:45.172 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:45.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:45.172 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T05:00:45.232 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-05-09T05:00:45.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:45.234 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-05-09T05:00:45.290 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-05-09T05:00:45.290 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T05:00:45.290 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T05:00:45.290 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T05:00:45.290 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T05:00:45.290 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:45.291 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T05:00:45.291 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T05:00:45.291 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T05:00:45.291 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T05:00:45.291 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T05:00:45.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:45.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:45.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:45.417 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-09T05:00:45.432 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-05-09T05:00:45.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:45.432 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list-id 2024-05-09T05:00:45.488 INFO:teuthology.orchestra.run.smithi114.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-05-09T05:00:45.488 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:45.488 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T05:00:45.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-09T05:00:45.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-05-09T05:00:45.488 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T05:00:45.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:45.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:45.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:45.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-05-09T05:00:45.630 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T05:00:45.630 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T05:00:45.630 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T05:00:45.630 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:45.630 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-05-09T05:00:45.630 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:45.630 INFO:teuthology.orchestra.run.smithi114.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T05:00:45.630 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether d2:84:2c:ba:b3:6b brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-09T05:00:45.631 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-09T05:00:45.631 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:45.631 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::d084:2cff:feba:b36b/64 scope link tentative 2024-05-09T05:00:45.631 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:45.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:45.632 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-05-09T05:00:45.696 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T05:00:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T05:00:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T05:00:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-05-09T05:00:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:45.697 INFO:teuthology.orchestra.run.smithi114.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T05:00:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether d2:84:2c:ba:b3:6b brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-09T05:00:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-09T05:00:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::d084:2cff:feba:b36b/64 scope link tentative 2024-05-09T05:00:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:00:45.698 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-05-09T05:00:45.698 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:45.698 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T05:00:45.698 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-05-09T05:00:45.698 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-05-09T05:00:45.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-05-09T05:00:45.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-05-09T05:00:45.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-05-09T05:00:45.699 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T05:00:45.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:45.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:45.850 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:00:45.851 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-05-09T05:00:45.851 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.1 up 2024-05-09T05:00:45.851 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set dev brx.1 master ceph-brx 2024-05-09T05:00:45.851 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-05-09T05:00:45.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:45.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:45.927 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T05:00:45.927 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-05-09T05:00:45.955 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-05-09T05:00:45.955 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T05:00:45.955 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-05-09T05:00:46.010 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-05-09T05:00:46.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:46.010 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T05:00:46.093 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T05:00:46.094 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T05:00:46.105 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T05:00:46.105 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T05:00:46.105 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.114:3300,172.21.15.114:3301" 2024-05-09T05:00:46.106 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with new device syntax: 3@0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2.cephfs=/ 2024-05-09T05:00:46.106 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.60:3300/172.21.15.114:3300/172.21.15.114:3301" will pass to kernel 2024-05-09T05:00:46.106 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-05-09T05:00:46.106 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-05-09T05:00:46.106 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-09T05:00:46.106 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-05-09T05:00:46.106 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-09T05:00:46.108 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-09T05:00:46.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:46.108 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-05-09T05:00:46.173 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T05:00:46.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:46 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T05:00:46.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:46.270 INFO:teuthology.orchestra.run.smithi114.stdout:client4716 2024-05-09T05:00:46.270 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T05:00:46.270 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:00:46.286 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T05:00:46.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:46 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T05:00:46.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:46.381 INFO:teuthology.orchestra.run.smithi114.stdout:client4716 2024-05-09T05:00:46.381 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T05:00:46.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:46 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T05:00:46.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:46.437 INFO:teuthology.orchestra.run.smithi114.stdout:client4716 2024-05-09T05:00:46.437 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2.client4716/status') 2024-05-09T05:00:46.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:00:46.495 INFO:teuthology.orchestra.run.smithi114.stdout:instance: client.4716 (3)172.21.15.114:0/2812239543 2024-05-09T05:00:46.495 INFO:teuthology.orchestra.run.smithi114.stdout:blocklisted: false 2024-05-09T05:00:46.495 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records in 2024-05-09T05:00:46.496 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records out 2024-05-09T05:00:46.496 INFO:teuthology.orchestra.run.smithi114.stderr:71 bytes copied, 4.0264e-05 s, 1.8 MB/s 2024-05-09T05:00:46.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-05-09T05:00:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:00:46.503 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-05-09T05:00:46.524 INFO:tasks.cephfs_test_runner:Starting test: test_backtrace (tasks.cephfs.test_backtrace.TestBacktrace) That the 'parent' 'layout' and 'symlink' xattrs on the head objects of files 2024-05-09T05:00:46.524 INFO:tasks.cephfs_test_runner:test_backtrace (tasks.cephfs.test_backtrace.TestBacktrace) 2024-05-09T05:00:46.525 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:00:46.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.641+0000 7f65e3714640 1 Processor -- start 2024-05-09T05:00:46.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.641+0000 7f65e3714640 1 -- start start 2024-05-09T05:00:46.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.642+0000 7f65e3714640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f65dc15f300 0x7f65dc072220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:46.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.642+0000 7f65e3714640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65dc06fe20 0x7f65dc0702a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:46.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.642+0000 7f65e3714640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f65dc0736d0 0x7f65dc073b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:46.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.642+0000 7f65e3714640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f65dc073240 con 0x7f65dc15f300 2024-05-09T05:00:46.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.642+0000 7f65e3714640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f65dc072760 con 0x7f65dc0736d0 2024-05-09T05:00:46.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.642+0000 7f65e3714640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f65dc0d08e0 con 0x7f65dc06fe20 2024-05-09T05:00:46.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.642+0000 7f65e14b0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f65dc15f300 0x7f65dc072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:46.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.642+0000 7f65e14b0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f65dc15f300 0x7f65dc072220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45996/0 (socket says 172.21.15.60:45996) 2024-05-09T05:00:46.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.642+0000 7f65e14b0640 1 -- 172.21.15.60:0/3849577152 learned_addr learned my addr 172.21.15.60:0/3849577152 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:46.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.642+0000 7f65e1cb1640 1 --2- 172.21.15.60:0/3849577152 >> v2:172.21.15.114:3300/0 conn(0x7f65dc0736d0 0x7f65dc073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:46.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.642+0000 7f65e0caf640 1 --2- 172.21.15.60:0/3849577152 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65dc06fe20 0x7f65dc0702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:46.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.643+0000 7f65e14b0640 1 -- 172.21.15.60:0/3849577152 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65dc06fe20 msgr2=0x7f65dc0702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:46.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.643+0000 7f65e14b0640 1 --2- 172.21.15.60:0/3849577152 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65dc06fe20 0x7f65dc0702a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:46.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.643+0000 7f65e14b0640 1 -- 172.21.15.60:0/3849577152 >> v2:172.21.15.114:3300/0 conn(0x7f65dc0736d0 msgr2=0x7f65dc073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:46.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.643+0000 7f65e14b0640 1 --2- 172.21.15.60:0/3849577152 >> v2:172.21.15.114:3300/0 conn(0x7f65dc0736d0 0x7f65dc073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:46.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.643+0000 7f65e14b0640 1 -- 172.21.15.60:0/3849577152 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65dc0d0bc0 con 0x7f65dc15f300 2024-05-09T05:00:46.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.643+0000 7f65e14b0640 1 --2- 172.21.15.60:0/3849577152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65dc15f300 0x7f65dc072220 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f65dc071160 tx=0x7f65d802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dee9e48d1f93ee8c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:46.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.644+0000 7f65d3fff640 1 -- 172.21.15.60:0/3849577152 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d8037070 con 0x7f65dc15f300 2024-05-09T05:00:46.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.644+0000 7f65d3fff640 1 -- 172.21.15.60:0/3849577152 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65d8032440 con 0x7f65dc15f300 2024-05-09T05:00:46.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.644+0000 7f65d3fff640 1 -- 172.21.15.60:0/3849577152 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d8032810 con 0x7f65dc15f300 2024-05-09T05:00:46.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.644+0000 7f65e3714640 1 -- 172.21.15.60:0/3849577152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65dc15f300 msgr2=0x7f65dc072220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:46.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.644+0000 7f65e3714640 1 --2- 172.21.15.60:0/3849577152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65dc15f300 0x7f65dc072220 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f65dc071160 tx=0x7f65d802a1e0 comp rx=0 tx=0).stop 2024-05-09T05:00:46.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.644+0000 7f65e3714640 1 -- 172.21.15.60:0/3849577152 shutdown_connections 2024-05-09T05:00:46.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.644+0000 7f65e3714640 1 --2- 172.21.15.60:0/3849577152 >> v2:172.21.15.114:3300/0 conn(0x7f65dc0736d0 0x7f65dc073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:46.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.644+0000 7f65e3714640 1 --2- 172.21.15.60:0/3849577152 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65dc06fe20 0x7f65dc0702a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:46.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.644+0000 7f65e3714640 1 --2- 172.21.15.60:0/3849577152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65dc15f300 0x7f65dc072220 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:46.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.644+0000 7f65e3714640 1 -- 172.21.15.60:0/3849577152 >> 172.21.15.60:0/3849577152 conn(0x7f65dc0b4ef0 msgr2=0x7f65dc06e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:46.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.644+0000 7f65e3714640 1 -- 172.21.15.60:0/3849577152 shutdown_connections 2024-05-09T05:00:46.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.644+0000 7f65e3714640 1 -- 172.21.15.60:0/3849577152 wait complete. 2024-05-09T05:00:46.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.646+0000 7f65e3714640 1 Processor -- start 2024-05-09T05:00:46.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.646+0000 7f65e3714640 1 -- start start 2024-05-09T05:00:46.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.646+0000 7f65e3714640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65dc06fe20 0x7f65dc1f6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:46.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.646+0000 7f65e3714640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65dc0736d0 0x7f65dc15afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:46.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.646+0000 7f65e3714640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65dc15f300 0x7f65dc0d4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:46.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.646+0000 7f65e3714640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f65dc09c1e0 con 0x7f65dc0736d0 2024-05-09T05:00:46.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.646+0000 7f65e3714640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f65dc074070 con 0x7f65dc06fe20 2024-05-09T05:00:46.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.646+0000 7f65e3714640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f65dc0d04f0 con 0x7f65dc15f300 2024-05-09T05:00:46.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.646+0000 7f65e0caf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65dc0736d0 0x7f65dc15afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:46.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.646+0000 7f65e0caf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65dc0736d0 0x7f65dc15afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46000/0 (socket says 172.21.15.60:46000) 2024-05-09T05:00:46.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.646+0000 7f65e0caf640 1 -- 172.21.15.60:0/2114804185 learned_addr learned my addr 172.21.15.60:0/2114804185 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:46.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.647+0000 7f65e14b0640 1 --2- 172.21.15.60:0/2114804185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65dc06fe20 0x7f65dc1f6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:46.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.647+0000 7f65e1cb1640 1 --2- 172.21.15.60:0/2114804185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65dc15f300 0x7f65dc0d4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:46.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.647+0000 7f65e0caf640 1 -- 172.21.15.60:0/2114804185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65dc15f300 msgr2=0x7f65dc0d4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:46.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.647+0000 7f65e0caf640 1 --2- 172.21.15.60:0/2114804185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65dc15f300 0x7f65dc0d4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:46.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.647+0000 7f65e0caf640 1 -- 172.21.15.60:0/2114804185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65dc06fe20 msgr2=0x7f65dc1f6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:46.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.647+0000 7f65e0caf640 1 --2- 172.21.15.60:0/2114804185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65dc06fe20 0x7f65dc1f6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:46.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.647+0000 7f65e0caf640 1 -- 172.21.15.60:0/2114804185 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65dc0d4de0 con 0x7f65dc0736d0 2024-05-09T05:00:46.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.647+0000 7f65e1cb1640 1 --2- 172.21.15.60:0/2114804185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65dc15f300 0x7f65dc0d4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:46.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.647+0000 7f65e0caf640 1 --2- 172.21.15.60:0/2114804185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65dc0736d0 0x7f65dc15afe0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f65cc00e8a0 tx=0x7f65cc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:46.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.647+0000 7f65d1ffb640 1 -- 172.21.15.60:0/2114804185 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc017070 con 0x7f65dc0736d0 2024-05-09T05:00:46.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.647+0000 7f65d1ffb640 1 -- 172.21.15.60:0/2114804185 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65cc00ee90 con 0x7f65dc0736d0 2024-05-09T05:00:46.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.647+0000 7f65d1ffb640 1 -- 172.21.15.60:0/2114804185 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc00cb80 con 0x7f65dc0736d0 2024-05-09T05:00:46.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.647+0000 7f65e3714640 1 -- 172.21.15.60:0/2114804185 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65dc0d50c0 con 0x7f65dc0736d0 2024-05-09T05:00:46.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.647+0000 7f65e3714640 1 -- 172.21.15.60:0/2114804185 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65dc1f5c10 con 0x7f65dc0736d0 2024-05-09T05:00:46.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.649+0000 7f65e3714640 1 -- 172.21.15.60:0/2114804185 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65dc051040 con 0x7f65dc0736d0 2024-05-09T05:00:46.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.649+0000 7f65d1ffb640 1 -- 172.21.15.60:0/2114804185 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f65cc01a070 con 0x7f65dc0736d0 2024-05-09T05:00:46.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.650+0000 7f65d1ffb640 1 --2- 172.21.15.60:0/2114804185 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f65c00762d0 0x7f65c0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:46.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.650+0000 7f65d1ffb640 1 -- 172.21.15.60:0/2114804185 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f65cc097b50 con 0x7f65dc0736d0 2024-05-09T05:00:46.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.650+0000 7f65e14b0640 1 --2- 172.21.15.60:0/2114804185 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f65c00762d0 0x7f65c0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:46.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.651+0000 7f65e14b0640 1 --2- 172.21.15.60:0/2114804185 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f65c00762d0 0x7f65c0078790 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f65d8034000 tx=0x7f65d802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:46.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.655+0000 7f65d1ffb640 1 -- 172.21.15.60:0/2114804185 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f65cc061160 con 0x7f65dc0736d0 2024-05-09T05:00:46.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.846+0000 7f65e3714640 1 -- 172.21.15.60:0/2114804185 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f65dc0d5290 con 0x7f65dc0736d0 2024-05-09T05:00:46.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.847+0000 7f65d1ffb640 1 -- 172.21.15.60:0/2114804185 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18490 (secure 0 0 0) 0x7f65cc060a60 con 0x7f65dc0736d0 2024-05-09T05:00:46.847 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:46.847 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:00:17.248958+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:00:10.358133+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:00:11.157769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T05:00:14.850459+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:00:46.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.849+0000 7f65e3714640 1 -- 172.21.15.60:0/2114804185 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f65c00762d0 msgr2=0x7f65c0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:46.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.849+0000 7f65e3714640 1 --2- 172.21.15.60:0/2114804185 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f65c00762d0 0x7f65c0078790 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f65d8034000 tx=0x7f65d802e000 comp rx=0 tx=0).stop 2024-05-09T05:00:46.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.849+0000 7f65e3714640 1 -- 172.21.15.60:0/2114804185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65dc0736d0 msgr2=0x7f65dc15afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:46.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.849+0000 7f65e3714640 1 --2- 172.21.15.60:0/2114804185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65dc0736d0 0x7f65dc15afe0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f65cc00e8a0 tx=0x7f65cc00ed60 comp rx=0 tx=0).stop 2024-05-09T05:00:46.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.849+0000 7f65e3714640 1 -- 172.21.15.60:0/2114804185 shutdown_connections 2024-05-09T05:00:46.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.849+0000 7f65e3714640 1 --2- 172.21.15.60:0/2114804185 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f65c00762d0 0x7f65c0078790 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:46.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.849+0000 7f65e3714640 1 --2- 172.21.15.60:0/2114804185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f65dc15f300 0x7f65dc0d4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:46.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.849+0000 7f65e3714640 1 --2- 172.21.15.60:0/2114804185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65dc0736d0 0x7f65dc15afe0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:46.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.849+0000 7f65e3714640 1 --2- 172.21.15.60:0/2114804185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65dc06fe20 0x7f65dc1f6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:46.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.849+0000 7f65e3714640 1 -- 172.21.15.60:0/2114804185 >> 172.21.15.60:0/2114804185 conn(0x7f65dc0b4ef0 msgr2=0x7f65dc0b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:46.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.850+0000 7f65e3714640 1 -- 172.21.15.60:0/2114804185 shutdown_connections 2024-05-09T05:00:46.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.850+0000 7f65e3714640 1 -- 172.21.15.60:0/2114804185 wait complete. 2024-05-09T05:00:46.860 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T05:00:06.925911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T05:00:10.358133+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T05:00:11.157769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T05:00:14.850459+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T05:00:46.861 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T05:00:46.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.967+0000 7f560be15640 1 Processor -- start 2024-05-09T05:00:46.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.968+0000 7f560be15640 1 -- start start 2024-05-09T05:00:46.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.968+0000 7f560be15640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f560406cf80 0x7f560415f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:46.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.968+0000 7f560be15640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56040cc990 con 0x7f56040b41d0 2024-05-09T05:00:46.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.968+0000 7f560be15640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5604160170 con 0x7f560415fce0 2024-05-09T05:00:46.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.969+0000 7f560be15640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f560406d3e0 con 0x7f560406cf80 2024-05-09T05:00:46.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.969+0000 7f5609bb1640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f56040b41d0 0x7f56040b45b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33240/0 (socket says 172.21.15.60:33240) 2024-05-09T05:00:46.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.969+0000 7f5609bb1640 1 -- 172.21.15.60:0/2472210813 learned_addr learned my addr 172.21.15.60:0/2472210813 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:46.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.969+0000 7f56093b0640 1 --2- 172.21.15.60:0/2472210813 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f560406cf80 0x7f560415f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:46.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.969+0000 7f5608baf640 1 -- 172.21.15.60:0/2472210813 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1877161633 0 0) 0x7f5600001110 con 0x7f56040b41d0 2024-05-09T05:00:46.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.969+0000 7f5608baf640 1 -- 172.21.15.60:0/2472210813 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55f0003520 con 0x7f56040b41d0 2024-05-09T05:00:46.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.969+0000 7f56093b0640 1 -- 172.21.15.60:0/2472210813 >> v1:172.21.15.114:6789/0 conn(0x7f560415fce0 legacy=0x7f56040d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:46.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.969+0000 7f5608baf640 1 -- 172.21.15.60:0/2472210813 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 467576656 0 0) 0x7f55fc001490 con 0x7f560415fce0 2024-05-09T05:00:46.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.969+0000 7f56093b0640 1 -- 172.21.15.60:0/2472210813 >> v1:172.21.15.60:6789/0 conn(0x7f56040b41d0 legacy=0x7f56040b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:46.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.969+0000 7f56093b0640 1 -- 172.21.15.60:0/2472210813 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56041603f0 con 0x7f560406cf80 2024-05-09T05:00:46.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.970+0000 7f56093b0640 1 --2- 172.21.15.60:0/2472210813 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f560406cf80 0x7f560415f7a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f55f40097d0 tx=0x7f55f402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfc78c2acc6bfb76 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:46.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.970+0000 7f5608baf640 1 -- 172.21.15.60:0/2472210813 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f4037070 con 0x7f560406cf80 2024-05-09T05:00:46.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.970+0000 7f5608baf640 1 -- 172.21.15.60:0/2472210813 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55f4032440 con 0x7f560406cf80 2024-05-09T05:00:46.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.970+0000 7f5608baf640 1 -- 172.21.15.60:0/2472210813 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f40327f0 con 0x7f560406cf80 2024-05-09T05:00:46.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.970+0000 7f560be15640 1 -- 172.21.15.60:0/2472210813 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f560406cf80 msgr2=0x7f560415f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:46.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.970+0000 7f560be15640 1 --2- 172.21.15.60:0/2472210813 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f560406cf80 0x7f560415f7a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f55f40097d0 tx=0x7f55f402a180 comp rx=0 tx=0).stop 2024-05-09T05:00:46.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.971+0000 7f560be15640 1 -- 172.21.15.60:0/2472210813 shutdown_connections 2024-05-09T05:00:46.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.971+0000 7f560be15640 1 --2- 172.21.15.60:0/2472210813 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f560406cf80 0x7f560415f7a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:46.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.971+0000 7f560be15640 1 -- 172.21.15.60:0/2472210813 >> 172.21.15.60:0/2472210813 conn(0x7f56040b3dc0 msgr2=0x7f5604070590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:46.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.971+0000 7f560be15640 1 -- 172.21.15.60:0/2472210813 shutdown_connections 2024-05-09T05:00:46.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.971+0000 7f560be15640 1 -- 172.21.15.60:0/2472210813 wait complete. 2024-05-09T05:00:46.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.972+0000 7f560be15640 1 Processor -- start 2024-05-09T05:00:46.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.972+0000 7f560be15640 1 -- start start 2024-05-09T05:00:46.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f560be15640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f560406cf80 0x7f56041f9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:46.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f560be15640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56040b41d0 0x7f560414f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:46.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f560be15640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f560415fce0 0x7f5604153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:46.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f560be15640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f560406caf0 con 0x7f560406cf80 2024-05-09T05:00:46.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f560be15640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f56041606d0 con 0x7f560415fce0 2024-05-09T05:00:46.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f560be15640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f560406c780 con 0x7f56040b41d0 2024-05-09T05:00:46.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f5609bb1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f560406cf80 0x7f56041f9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:46.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f5609bb1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f560406cf80 0x7f56041f9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46004/0 (socket says 172.21.15.60:46004) 2024-05-09T05:00:46.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f5609bb1640 1 -- 172.21.15.60:0/2914732548 learned_addr learned my addr 172.21.15.60:0/2914732548 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:46.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f56093b0640 1 --2- 172.21.15.60:0/2914732548 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56040b41d0 0x7f560414f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:46.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f560a3b2640 1 --2- 172.21.15.60:0/2914732548 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f560415fce0 0x7f5604153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:46.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f5609bb1640 1 -- 172.21.15.60:0/2914732548 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56040b41d0 msgr2=0x7f560414f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:46.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f5609bb1640 1 --2- 172.21.15.60:0/2914732548 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56040b41d0 0x7f560414f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:46.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f5609bb1640 1 -- 172.21.15.60:0/2914732548 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f560415fce0 msgr2=0x7f5604153440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:46.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f5609bb1640 1 --2- 172.21.15.60:0/2914732548 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f560415fce0 0x7f5604153440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:46.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f5609bb1640 1 -- 172.21.15.60:0/2914732548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f560414fd90 con 0x7f560406cf80 2024-05-09T05:00:46.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f5609bb1640 1 --2- 172.21.15.60:0/2914732548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f560406cf80 0x7f56041f9090 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f560000c730 tx=0x7f560000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:46.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f55fa7fc640 1 -- 172.21.15.60:0/2914732548 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5600017070 con 0x7f560406cf80 2024-05-09T05:00:46.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f55fa7fc640 1 -- 172.21.15.60:0/2914732548 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5600001d30 con 0x7f560406cf80 2024-05-09T05:00:46.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f560be15640 1 -- 172.21.15.60:0/2914732548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5604150070 con 0x7f560406cf80 2024-05-09T05:00:46.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f55fa7fc640 1 -- 172.21.15.60:0/2914732548 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5600005cb0 con 0x7f560406cf80 2024-05-09T05:00:46.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.973+0000 7f560be15640 1 -- 172.21.15.60:0/2914732548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5604150630 con 0x7f560406cf80 2024-05-09T05:00:46.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.974+0000 7f560be15640 1 -- 172.21.15.60:0/2914732548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56040d1220 con 0x7f560406cf80 2024-05-09T05:00:46.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.975+0000 7f55fa7fc640 1 -- 172.21.15.60:0/2914732548 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f560001a070 con 0x7f560406cf80 2024-05-09T05:00:46.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.975+0000 7f55fa7fc640 1 --2- 172.21.15.60:0/2914732548 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f55e80762d0 0x7f55e8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:46.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.976+0000 7f55fa7fc640 1 -- 172.21.15.60:0/2914732548 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5600096b50 con 0x7f560406cf80 2024-05-09T05:00:46.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.976+0000 7f56093b0640 1 --2- 172.21.15.60:0/2914732548 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f55e80762d0 0x7f55e8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:46.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.976+0000 7f56093b0640 1 --2- 172.21.15.60:0/2914732548 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f55e80762d0 0x7f55e8078790 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f55f4034000 tx=0x7f55f402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:46.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:46.980+0000 7f55fa7fc640 1 -- 172.21.15.60:0/2914732548 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f56040cc990 con 0x7f560406cf80 2024-05-09T05:00:47.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.172+0000 7f560be15640 1 -- 172.21.15.60:0/2914732548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f560406d9c0 con 0x7f560406cf80 2024-05-09T05:00:47.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.172+0000 7f55fa7fc640 1 -- 172.21.15.60:0/2914732548 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f5600068150 con 0x7f560406cf80 2024-05-09T05:00:47.172 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-05-09T05:00:47.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.174+0000 7f560be15640 1 -- 172.21.15.60:0/2914732548 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f55e80762d0 msgr2=0x7f55e8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:47.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.174+0000 7f560be15640 1 --2- 172.21.15.60:0/2914732548 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f55e80762d0 0x7f55e8078790 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f55f4034000 tx=0x7f55f402e000 comp rx=0 tx=0).stop 2024-05-09T05:00:47.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.174+0000 7f560be15640 1 -- 172.21.15.60:0/2914732548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f560406cf80 msgr2=0x7f56041f9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:47.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.174+0000 7f560be15640 1 --2- 172.21.15.60:0/2914732548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f560406cf80 0x7f56041f9090 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f560000c730 tx=0x7f560000cbf0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.174+0000 7f560be15640 1 -- 172.21.15.60:0/2914732548 shutdown_connections 2024-05-09T05:00:47.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.174+0000 7f560be15640 1 --2- 172.21.15.60:0/2914732548 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f55e80762d0 0x7f55e8078790 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.174+0000 7f560be15640 1 --2- 172.21.15.60:0/2914732548 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f560415fce0 0x7f5604153440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.174+0000 7f560be15640 1 --2- 172.21.15.60:0/2914732548 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f56040b41d0 0x7f560414f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.174+0000 7f560be15640 1 --2- 172.21.15.60:0/2914732548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f560406cf80 0x7f56041f9090 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.174+0000 7f560be15640 1 -- 172.21.15.60:0/2914732548 >> 172.21.15.60:0/2914732548 conn(0x7f56040b3dc0 msgr2=0x7f5604162220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:47.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.174+0000 7f560be15640 1 -- 172.21.15.60:0/2914732548 shutdown_connections 2024-05-09T05:00:47.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.174+0000 7f560be15640 1 -- 172.21.15.60:0/2914732548 wait complete. 2024-05-09T05:00:47.186 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T05:00:47.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.292+0000 7fd94f959640 1 Processor -- start 2024-05-09T05:00:47.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.292+0000 7fd94f959640 1 -- start start 2024-05-09T05:00:47.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.293+0000 7fd94f959640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd9481486d0 0x7fd9480b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.293+0000 7fd94f959640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd948148c40 0x7fd9480d2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.293+0000 7fd94f959640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9480b6400 con 0x7fd9480b7080 2024-05-09T05:00:47.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.293+0000 7fd94f959640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fd948071810 con 0x7fd9481486d0 2024-05-09T05:00:47.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.293+0000 7fd94f959640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd948071a60 con 0x7fd948148c40 2024-05-09T05:00:47.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.293+0000 7fd94d6f5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd9481486d0 0x7fd9480b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.293+0000 7fd94d6f5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd9481486d0 0x7fd9480b5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46010/0 (socket says 172.21.15.60:46010) 2024-05-09T05:00:47.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.293+0000 7fd94d6f5640 1 -- 172.21.15.60:0/2135594506 learned_addr learned my addr 172.21.15.60:0/2135594506 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:47.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.294+0000 7fd94cef4640 1 --2- 172.21.15.60:0/2135594506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd948148c40 0x7fd9480d2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.294+0000 7fd94d6f5640 1 -- 172.21.15.60:0/2135594506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd948148c40 msgr2=0x7fd9480d2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:47.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.294+0000 7fd94d6f5640 1 --2- 172.21.15.60:0/2135594506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd948148c40 0x7fd9480d2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.294+0000 7fd94d6f5640 1 -- 172.21.15.60:0/2135594506 >> v1:172.21.15.114:6789/0 conn(0x7fd9480b7080 legacy=0x7fd948072820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:47.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.294+0000 7fd94d6f5640 1 -- 172.21.15.60:0/2135594506 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd948071d40 con 0x7fd9481486d0 2024-05-09T05:00:47.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.294+0000 7fd94cef4640 1 --2- 172.21.15.60:0/2135594506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd948148c40 0x7fd9480d2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:47.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.294+0000 7fd94d6f5640 1 --2- 172.21.15.60:0/2135594506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9481486d0 0x7fd9480b5ec0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd940009870 tx=0x7fd94002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=277537d2e373c7c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:47.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.294+0000 7fd947fff640 1 -- 172.21.15.60:0/2135594506 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd940037070 con 0x7fd9481486d0 2024-05-09T05:00:47.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.294+0000 7fd947fff640 1 -- 172.21.15.60:0/2135594506 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd940032440 con 0x7fd9481486d0 2024-05-09T05:00:47.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.295+0000 7fd947fff640 1 -- 172.21.15.60:0/2135594506 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd940032810 con 0x7fd9481486d0 2024-05-09T05:00:47.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.295+0000 7fd94f959640 1 -- 172.21.15.60:0/2135594506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9481486d0 msgr2=0x7fd9480b5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:47.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.295+0000 7fd94f959640 1 --2- 172.21.15.60:0/2135594506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9481486d0 0x7fd9480b5ec0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd940009870 tx=0x7fd94002a250 comp rx=0 tx=0).stop 2024-05-09T05:00:47.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.295+0000 7fd94f959640 1 -- 172.21.15.60:0/2135594506 shutdown_connections 2024-05-09T05:00:47.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.295+0000 7fd94f959640 1 --2- 172.21.15.60:0/2135594506 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd948148c40 0x7fd9480d2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.295+0000 7fd94f959640 1 --2- 172.21.15.60:0/2135594506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9481486d0 0x7fd9480b5ec0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.295+0000 7fd94f959640 1 -- 172.21.15.60:0/2135594506 >> 172.21.15.60:0/2135594506 conn(0x7fd94806f6f0 msgr2=0x7fd9480ba5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:47.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.295+0000 7fd94f959640 1 -- 172.21.15.60:0/2135594506 shutdown_connections 2024-05-09T05:00:47.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.295+0000 7fd94f959640 1 -- 172.21.15.60:0/2135594506 wait complete. 2024-05-09T05:00:47.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.296+0000 7fd94f959640 1 Processor -- start 2024-05-09T05:00:47.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.296+0000 7fd94f959640 1 -- start start 2024-05-09T05:00:47.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.297+0000 7fd94f959640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9480b7080 0x7fd94815fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.297+0000 7fd94f959640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9481486d0 0x7fd94815e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.297+0000 7fd94f959640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd948148c40 0x7fd948136340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.297+0000 7fd94f959640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd9480d31d0 con 0x7fd9480b7080 2024-05-09T05:00:47.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.297+0000 7fd94f959640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd948072300 con 0x7fd948148c40 2024-05-09T05:00:47.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.297+0000 7fd94f959640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd9480d3350 con 0x7fd9481486d0 2024-05-09T05:00:47.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.297+0000 7fd94d6f5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9480b7080 0x7fd94815fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.297+0000 7fd94d6f5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9480b7080 0x7fd94815fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46012/0 (socket says 172.21.15.60:46012) 2024-05-09T05:00:47.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.297+0000 7fd94d6f5640 1 -- 172.21.15.60:0/435332925 learned_addr learned my addr 172.21.15.60:0/435332925 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:47.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.297+0000 7fd94cef4640 1 --2- 172.21.15.60:0/435332925 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9481486d0 0x7fd94815e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.297+0000 7fd94d6f5640 1 -- 172.21.15.60:0/435332925 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9481486d0 msgr2=0x7fd94815e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:47.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.297+0000 7fd94def6640 1 --2- 172.21.15.60:0/435332925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd948148c40 0x7fd948136340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.297+0000 7fd94d6f5640 1 --2- 172.21.15.60:0/435332925 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9481486d0 0x7fd94815e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.297+0000 7fd94d6f5640 1 -- 172.21.15.60:0/435332925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd948148c40 msgr2=0x7fd948136340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:47.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.297+0000 7fd94d6f5640 1 --2- 172.21.15.60:0/435332925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd948148c40 0x7fd948136340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.297+0000 7fd94d6f5640 1 -- 172.21.15.60:0/435332925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9481eb6e0 con 0x7fd9480b7080 2024-05-09T05:00:47.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.297+0000 7fd94def6640 1 --2- 172.21.15.60:0/435332925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd948148c40 0x7fd948136340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:47.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.298+0000 7fd94d6f5640 1 --2- 172.21.15.60:0/435332925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9480b7080 0x7fd94815fcb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd940009870 tx=0x7fd9400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:47.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.298+0000 7fd945ffb640 1 -- 172.21.15.60:0/435332925 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd940041020 con 0x7fd9480b7080 2024-05-09T05:00:47.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.298+0000 7fd945ffb640 1 -- 172.21.15.60:0/435332925 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd940030970 con 0x7fd9480b7080 2024-05-09T05:00:47.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.298+0000 7fd94f959640 1 -- 172.21.15.60:0/435332925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd948160340 con 0x7fd9480b7080 2024-05-09T05:00:47.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.298+0000 7fd945ffb640 1 -- 172.21.15.60:0/435332925 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd940037040 con 0x7fd9480b7080 2024-05-09T05:00:47.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.298+0000 7fd94f959640 1 -- 172.21.15.60:0/435332925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd948160900 con 0x7fd9480b7080 2024-05-09T05:00:47.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.299+0000 7fd94f959640 1 -- 172.21.15.60:0/435332925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd948051040 con 0x7fd9480b7080 2024-05-09T05:00:47.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.300+0000 7fd945ffb640 1 -- 172.21.15.60:0/435332925 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd940044020 con 0x7fd9480b7080 2024-05-09T05:00:47.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.300+0000 7fd945ffb640 1 --2- 172.21.15.60:0/435332925 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd9340762d0 0x7fd934078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.300+0000 7fd945ffb640 1 -- 172.21.15.60:0/435332925 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd9400b8320 con 0x7fd9480b7080 2024-05-09T05:00:47.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.301+0000 7fd94cef4640 1 --2- 172.21.15.60:0/435332925 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd9340762d0 0x7fd934078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.301+0000 7fd94cef4640 1 --2- 172.21.15.60:0/435332925 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd9340762d0 0x7fd934078790 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd948162dd0 tx=0x7fd93c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:47.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.305+0000 7fd945ffb640 1 -- 172.21.15.60:0/435332925 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd94004b030 con 0x7fd9480b7080 2024-05-09T05:00:47.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.493+0000 7fd94f959640 1 -- 172.21.15.60:0/435332925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd948073980 con 0x7fd9480b7080 2024-05-09T05:00:47.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.494+0000 7fd945ffb640 1 -- 172.21.15.60:0/435332925 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7fd9480b6400 con 0x7fd9480b7080 2024-05-09T05:00:47.493 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-05-09T05:00:47.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.496+0000 7fd94f959640 1 -- 172.21.15.60:0/435332925 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd9340762d0 msgr2=0x7fd934078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:47.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.496+0000 7fd94f959640 1 --2- 172.21.15.60:0/435332925 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd9340762d0 0x7fd934078790 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd948162dd0 tx=0x7fd93c008040 comp rx=0 tx=0).stop 2024-05-09T05:00:47.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.496+0000 7fd94f959640 1 -- 172.21.15.60:0/435332925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9480b7080 msgr2=0x7fd94815fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:47.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.496+0000 7fd94f959640 1 --2- 172.21.15.60:0/435332925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9480b7080 0x7fd94815fcb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd940009870 tx=0x7fd9400046c0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.496+0000 7fd94f959640 1 -- 172.21.15.60:0/435332925 shutdown_connections 2024-05-09T05:00:47.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.496+0000 7fd94f959640 1 --2- 172.21.15.60:0/435332925 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd9340762d0 0x7fd934078790 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.496+0000 7fd94f959640 1 --2- 172.21.15.60:0/435332925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd948148c40 0x7fd948136340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.496+0000 7fd94f959640 1 --2- 172.21.15.60:0/435332925 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9481486d0 0x7fd94815e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.496+0000 7fd94f959640 1 --2- 172.21.15.60:0/435332925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9480b7080 0x7fd94815fcb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.496+0000 7fd94f959640 1 -- 172.21.15.60:0/435332925 >> 172.21.15.60:0/435332925 conn(0x7fd94806f6f0 msgr2=0x7fd9480d3940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:47.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.496+0000 7fd94f959640 1 -- 172.21.15.60:0/435332925 shutdown_connections 2024-05-09T05:00:47.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.496+0000 7fd94f959640 1 -- 172.21.15.60:0/435332925 wait complete. 2024-05-09T05:00:47.506 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T05:00:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.613+0000 7f92638dc640 1 Processor -- start 2024-05-09T05:00:47.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.613+0000 7f92638dc640 1 -- start start 2024-05-09T05:00:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.614+0000 7f92638dc640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f925c15f330 0x7f925c0b7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.614+0000 7f92638dc640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f925c072c00 con 0x7f925c162fa0 2024-05-09T05:00:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.614+0000 7f92638dc640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f925c163540 con 0x7f925c0b65d0 2024-05-09T05:00:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.614+0000 7f92638dc640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f925c163820 con 0x7f925c15f330 2024-05-09T05:00:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.614+0000 7f9260e77640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f925c162fa0 0x7f925c0b5ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33254/0 (socket says 172.21.15.60:33254) 2024-05-09T05:00:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.614+0000 7f9260e77640 1 -- 172.21.15.60:0/716342185 learned_addr learned my addr 172.21.15.60:0/716342185 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.614+0000 7f925bfff640 1 -- 172.21.15.60:0/716342185 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 372901281 0 0) 0x7f925c072c00 con 0x7f925c162fa0 2024-05-09T05:00:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.614+0000 7f9261678640 1 --2- 172.21.15.60:0/716342185 >> v2:172.21.15.114:3300/0 conn(0x7f925c15f330 0x7f925c0b7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f925bfff640 1 -- 172.21.15.60:0/716342185 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9244003520 con 0x7f925c162fa0 2024-05-09T05:00:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f925bfff640 1 -- 172.21.15.60:0/716342185 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 692983215 0 0) 0x7f9244003520 con 0x7f925c162fa0 2024-05-09T05:00:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f925bfff640 1 -- 172.21.15.60:0/716342185 >> v2:172.21.15.114:3300/0 conn(0x7f925c15f330 msgr2=0x7f925c0b7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f925bfff640 1 --2- 172.21.15.60:0/716342185 >> v2:172.21.15.114:3300/0 conn(0x7f925c15f330 0x7f925c0b7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f925bfff640 1 -- 172.21.15.60:0/716342185 >> v1:172.21.15.114:6789/0 conn(0x7f925c0b65d0 legacy=0x7f925c0730e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f925bfff640 1 -- 172.21.15.60:0/716342185 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925c163b00 con 0x7f925c162fa0 2024-05-09T05:00:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f925bfff640 1 -- 172.21.15.60:0/716342185 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f9250001710 con 0x7f925c162fa0 2024-05-09T05:00:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f925bfff640 1 -- 172.21.15.60:0/716342185 >> v1:172.21.15.60:6789/0 conn(0x7f925c162fa0 legacy=0x7f925c0b5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f925bfff640 1 --2- 172.21.15.60:0/716342185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9244003a50 0x7f9244003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f925bfff640 1 --2- 172.21.15.60:0/716342185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9244004800 0x7f9244004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f925bfff640 1 --2- 172.21.15.60:0/716342185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92440051c0 0x7f924400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f925bfff640 1 -- 172.21.15.60:0/716342185 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f925c163820 con 0x7f9244004800 2024-05-09T05:00:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f925bfff640 1 -- 172.21.15.60:0/716342185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f924400c160 con 0x7f92440051c0 2024-05-09T05:00:47.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f925bfff640 1 -- 172.21.15.60:0/716342185 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f924400c360 con 0x7f9244003a50 2024-05-09T05:00:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f9260e77640 1 --2- 172.21.15.60:0/716342185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9244004800 0x7f9244004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f9261678640 1 --2- 172.21.15.60:0/716342185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9244003a50 0x7f9244003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f9261e79640 1 --2- 172.21.15.60:0/716342185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92440051c0 0x7f924400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f9260e77640 1 -- 172.21.15.60:0/716342185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9244003a50 msgr2=0x7f9244003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f9260e77640 1 --2- 172.21.15.60:0/716342185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9244003a50 0x7f9244003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f9260e77640 1 -- 172.21.15.60:0/716342185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92440051c0 msgr2=0x7f924400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f9260e77640 1 --2- 172.21.15.60:0/716342185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92440051c0 0x7f924400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f9261e79640 1 -- 172.21.15.60:0/716342185 reap_dead start 2024-05-09T05:00:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.615+0000 7f9260e77640 1 -- 172.21.15.60:0/716342185 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9244003520 con 0x7f9244004800 2024-05-09T05:00:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.616+0000 7f9260e77640 1 --2- 172.21.15.60:0/716342185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9244004800 0x7f9244004c80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f925000c2f0 tx=0x7f9244005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fa7a567f01a6a2f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.616+0000 7f925bfff640 1 -- 172.21.15.60:0/716342185 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9250007d90 con 0x7f9244004800 2024-05-09T05:00:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.616+0000 7f925bfff640 1 -- 172.21.15.60:0/716342185 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f925002e070 con 0x7f9244004800 2024-05-09T05:00:47.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.616+0000 7f925bfff640 1 -- 172.21.15.60:0/716342185 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9250005d10 con 0x7f9244004800 2024-05-09T05:00:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.617+0000 7f92638dc640 1 -- 172.21.15.60:0/716342185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9244004800 msgr2=0x7f9244004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.617+0000 7f92638dc640 1 --2- 172.21.15.60:0/716342185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9244004800 0x7f9244004c80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f925000c2f0 tx=0x7f9244005830 comp rx=0 tx=0).stop 2024-05-09T05:00:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.617+0000 7f92638dc640 1 -- 172.21.15.60:0/716342185 shutdown_connections 2024-05-09T05:00:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.617+0000 7f92638dc640 1 --2- 172.21.15.60:0/716342185 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9244004800 0x7f9244004c80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.617+0000 7f92638dc640 1 -- 172.21.15.60:0/716342185 >> 172.21.15.60:0/716342185 conn(0x7f925c06ca80 msgr2=0x7f925c0d53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.617+0000 7f92638dc640 1 -- 172.21.15.60:0/716342185 shutdown_connections 2024-05-09T05:00:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.617+0000 7f92638dc640 1 -- 172.21.15.60:0/716342185 wait complete. 2024-05-09T05:00:47.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.618+0000 7f92638dc640 1 Processor -- start 2024-05-09T05:00:47.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.618+0000 7f92638dc640 1 -- start start 2024-05-09T05:00:47.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.619+0000 7f92638dc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9244004800 0x7f925c1f4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.619+0000 7f92638dc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f925c160640 0x7f925c160fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.619+0000 7f92638dc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f925c15d490 0x7f925c15d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.619+0000 7f92638dc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f925c15f710 con 0x7f925c15d490 2024-05-09T05:00:47.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.619+0000 7f92638dc640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f925c163de0 con 0x7f925c160640 2024-05-09T05:00:47.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.619+0000 7f92638dc640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f925c1f4b50 con 0x7f9244004800 2024-05-09T05:00:47.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.619+0000 7f9261e79640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f925c15d490 0x7f925c15d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.619+0000 7f9261e79640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f925c15d490 0x7f925c15d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46028/0 (socket says 172.21.15.60:46028) 2024-05-09T05:00:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.619+0000 7f9261e79640 1 -- 172.21.15.60:0/3599075031 learned_addr learned my addr 172.21.15.60:0/3599075031 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.619+0000 7f9260e77640 1 --2- 172.21.15.60:0/3599075031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f925c160640 0x7f925c160fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.619+0000 7f9261678640 1 --2- 172.21.15.60:0/3599075031 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9244004800 0x7f925c1f4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.619+0000 7f9261e79640 1 -- 172.21.15.60:0/3599075031 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9244004800 msgr2=0x7f925c1f4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.619+0000 7f9261e79640 1 --2- 172.21.15.60:0/3599075031 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9244004800 0x7f925c1f4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.619+0000 7f9261e79640 1 -- 172.21.15.60:0/3599075031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f925c160640 msgr2=0x7f925c160fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.619+0000 7f9261e79640 1 --2- 172.21.15.60:0/3599075031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f925c160640 0x7f925c160fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.619+0000 7f9261e79640 1 -- 172.21.15.60:0/3599075031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925c15bd20 con 0x7f925c15d490 2024-05-09T05:00:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.619+0000 7f9261e79640 1 --2- 172.21.15.60:0/3599075031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f925c15d490 0x7f925c15d910 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f925c0b70c0 tx=0x7f924c00fbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.620+0000 7f9259ffb640 1 -- 172.21.15.60:0/3599075031 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924c019070 con 0x7f925c15d490 2024-05-09T05:00:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.620+0000 7f9259ffb640 1 -- 172.21.15.60:0/3599075031 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f924c006d60 con 0x7f925c15d490 2024-05-09T05:00:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.620+0000 7f92638dc640 1 -- 172.21.15.60:0/3599075031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f925c15b1a0 con 0x7f925c15d490 2024-05-09T05:00:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.620+0000 7f9259ffb640 1 -- 172.21.15.60:0/3599075031 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924c003bb0 con 0x7f925c15d490 2024-05-09T05:00:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.620+0000 7f92638dc640 1 -- 172.21.15.60:0/3599075031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f925c161510 con 0x7f925c15d490 2024-05-09T05:00:47.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.621+0000 7f92638dc640 1 -- 172.21.15.60:0/3599075031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f925c15b6c0 con 0x7f925c15d490 2024-05-09T05:00:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.621+0000 7f9259ffb640 1 -- 172.21.15.60:0/3599075031 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f924c01c070 con 0x7f925c15d490 2024-05-09T05:00:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.622+0000 7f9259ffb640 1 --2- 172.21.15.60:0/3599075031 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9248076300 0x7f92480787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.622+0000 7f9259ffb640 1 -- 172.21.15.60:0/3599075031 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f924c096990 con 0x7f925c15d490 2024-05-09T05:00:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.623+0000 7f9261678640 1 --2- 172.21.15.60:0/3599075031 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9248076300 0x7f92480787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.623+0000 7f9261678640 1 --2- 172.21.15.60:0/3599075031 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9248076300 0x7f92480787c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9254009c40 tx=0x7f9254009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:47.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.627+0000 7f9259ffb640 1 -- 172.21.15.60:0/3599075031 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f924c05f8c0 con 0x7f925c15d490 2024-05-09T05:00:47.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.815+0000 7f92638dc640 1 -- 172.21.15.60:0/3599075031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f925c051040 con 0x7f925c15d490 2024-05-09T05:00:47.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.816+0000 7f9259ffb640 1 -- 172.21.15.60:0/3599075031 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f924c063bc0 con 0x7f925c15d490 2024-05-09T05:00:47.815 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2024-05-09T05:00:47.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.818+0000 7f92638dc640 1 -- 172.21.15.60:0/3599075031 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9248076300 msgr2=0x7f92480787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.818+0000 7f92638dc640 1 --2- 172.21.15.60:0/3599075031 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9248076300 0x7f92480787c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9254009c40 tx=0x7f9254009190 comp rx=0 tx=0).stop 2024-05-09T05:00:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.818+0000 7f92638dc640 1 -- 172.21.15.60:0/3599075031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f925c15d490 msgr2=0x7f925c15d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.818+0000 7f92638dc640 1 --2- 172.21.15.60:0/3599075031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f925c15d490 0x7f925c15d910 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f925c0b70c0 tx=0x7f924c00fbd0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.818+0000 7f92638dc640 1 -- 172.21.15.60:0/3599075031 shutdown_connections 2024-05-09T05:00:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.818+0000 7f92638dc640 1 --2- 172.21.15.60:0/3599075031 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9248076300 0x7f92480787c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.818+0000 7f92638dc640 1 --2- 172.21.15.60:0/3599075031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f925c15d490 0x7f925c15d910 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.818+0000 7f92638dc640 1 --2- 172.21.15.60:0/3599075031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f925c160640 0x7f925c160fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.818+0000 7f92638dc640 1 --2- 172.21.15.60:0/3599075031 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9244004800 0x7f925c1f4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.818+0000 7f92638dc640 1 -- 172.21.15.60:0/3599075031 >> 172.21.15.60:0/3599075031 conn(0x7f925c06ca80 msgr2=0x7f925c0b8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.818+0000 7f92638dc640 1 -- 172.21.15.60:0/3599075031 shutdown_connections 2024-05-09T05:00:47.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.818+0000 7f92638dc640 1 -- 172.21.15.60:0/3599075031 wait complete. 2024-05-09T05:00:47.828 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T05:00:47.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.936+0000 7f67f1cda640 1 Processor -- start 2024-05-09T05:00:47.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.936+0000 7f67f1cda640 1 -- start start 2024-05-09T05:00:47.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.937+0000 7f67f1cda640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f67ec1486d0 0x7f67ec0b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.937+0000 7f67f1cda640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67ec0b7080 0x7f67ec0b74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.937+0000 7f67f1cda640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67ec0b6400 con 0x7f67ec148c40 2024-05-09T05:00:47.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.937+0000 7f67f1cda640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f67ec0b7a10 con 0x7f67ec1486d0 2024-05-09T05:00:47.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.937+0000 7f67f1cda640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f67ec071810 con 0x7f67ec0b7080 2024-05-09T05:00:47.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.937+0000 7f67ebfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67ec0b7080 0x7f67ec0b74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.937+0000 7f67eb7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f67ec1486d0 0x7f67ec0b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.937+0000 7f67ebfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67ec0b7080 0x7f67ec0b74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.60:47526/0 (socket says 172.21.15.60:47526) 2024-05-09T05:00:47.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.937+0000 7f67eb7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f67ec1486d0 0x7f67ec0b5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.60:42564/0 (socket says 172.21.15.60:42564) 2024-05-09T05:00:47.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.937+0000 7f67ebfff640 1 -- 172.21.15.60:0/519170647 learned_addr learned my addr 172.21.15.60:0/519170647 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:47.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.937+0000 7f67eb7fe640 1 -- 172.21.15.60:0/519170647 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67ec0b7080 msgr2=0x7f67ec0b74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:47.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.937+0000 7f67eb7fe640 1 --2- 172.21.15.60:0/519170647 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67ec0b7080 0x7f67ec0b74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.937+0000 7f67eb7fe640 1 -- 172.21.15.60:0/519170647 >> v1:172.21.15.114:6789/0 conn(0x7f67ec148c40 legacy=0x7f67ec0d2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:47.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.937+0000 7f67eaffd640 1 -- 172.21.15.60:0/519170647 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1861677456 0 0) 0x7f67d4001110 con 0x7f67ec148c40 2024-05-09T05:00:47.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.937+0000 7f67eb7fe640 1 -- 172.21.15.60:0/519170647 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67ec0d33a0 con 0x7f67ec1486d0 2024-05-09T05:00:47.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.937+0000 7f67ebfff640 1 --2- 172.21.15.60:0/519170647 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67ec0b7080 0x7f67ec0b74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:00:47.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.938+0000 7f67eb7fe640 1 --2- 172.21.15.60:0/519170647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ec1486d0 0x7f67ec0b5ec0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0097a0 tx=0x7f67dc02a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a0049d929e20ee2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:47.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.938+0000 7f67eaffd640 1 -- 172.21.15.60:0/519170647 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc037070 con 0x7f67ec1486d0 2024-05-09T05:00:47.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.938+0000 7f67eaffd640 1 -- 172.21.15.60:0/519170647 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67dc032440 con 0x7f67ec1486d0 2024-05-09T05:00:47.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.938+0000 7f67eaffd640 1 -- 172.21.15.60:0/519170647 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc032810 con 0x7f67ec1486d0 2024-05-09T05:00:47.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.939+0000 7f67f1cda640 1 -- 172.21.15.60:0/519170647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ec1486d0 msgr2=0x7f67ec0b5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:47.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.939+0000 7f67f1cda640 1 --2- 172.21.15.60:0/519170647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ec1486d0 0x7f67ec0b5ec0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0097a0 tx=0x7f67dc02a210 comp rx=0 tx=0).stop 2024-05-09T05:00:47.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.939+0000 7f67f1cda640 1 -- 172.21.15.60:0/519170647 shutdown_connections 2024-05-09T05:00:47.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.939+0000 7f67f1cda640 1 --2- 172.21.15.60:0/519170647 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67ec0b7080 0x7f67ec0b74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.939+0000 7f67f1cda640 1 --2- 172.21.15.60:0/519170647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ec1486d0 0x7f67ec0b5ec0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.939+0000 7f67f1cda640 1 -- 172.21.15.60:0/519170647 >> 172.21.15.60:0/519170647 conn(0x7f67ec06f6f0 msgr2=0x7f67ec0ba5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:47.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.939+0000 7f67f1cda640 1 -- 172.21.15.60:0/519170647 shutdown_connections 2024-05-09T05:00:47.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.939+0000 7f67f1cda640 1 -- 172.21.15.60:0/519170647 wait complete. 2024-05-09T05:00:47.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.940+0000 7f67f1cda640 1 Processor -- start 2024-05-09T05:00:47.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.940+0000 7f67f1cda640 1 -- start start 2024-05-09T05:00:47.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.941+0000 7f67f1cda640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67ec0b7080 0x7f67ec143b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.941+0000 7f67f1cda640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ec1486d0 0x7f67ec146b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.941+0000 7f67f1cda640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67ec148c40 0x7f67ec147080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.941+0000 7f67f1cda640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f67ec072020 con 0x7f67ec148c40 2024-05-09T05:00:47.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.941+0000 7f67f1cda640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f67ec071d40 con 0x7f67ec1486d0 2024-05-09T05:00:47.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.941+0000 7f67f1cda640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f67ec072300 con 0x7f67ec0b7080 2024-05-09T05:00:47.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.941+0000 7f67ebfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67ec148c40 0x7f67ec147080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.941+0000 7f67ebfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67ec148c40 0x7f67ec147080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46034/0 (socket says 172.21.15.60:46034) 2024-05-09T05:00:47.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.941+0000 7f67ebfff640 1 -- 172.21.15.60:0/3150373391 learned_addr learned my addr 172.21.15.60:0/3150373391 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:47.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.941+0000 7f67eb7fe640 1 --2- 172.21.15.60:0/3150373391 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67ec0b7080 0x7f67ec143b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.941+0000 7f67e3fff640 1 --2- 172.21.15.60:0/3150373391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ec1486d0 0x7f67ec146b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.942+0000 7f67ebfff640 1 -- 172.21.15.60:0/3150373391 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67ec0b7080 msgr2=0x7f67ec143b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:47.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.942+0000 7f67ebfff640 1 --2- 172.21.15.60:0/3150373391 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67ec0b7080 0x7f67ec143b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.942+0000 7f67ebfff640 1 -- 172.21.15.60:0/3150373391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ec1486d0 msgr2=0x7f67ec146b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:47.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.942+0000 7f67ebfff640 1 --2- 172.21.15.60:0/3150373391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ec1486d0 0x7f67ec146b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:47.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.942+0000 7f67ebfff640 1 -- 172.21.15.60:0/3150373391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67ec161420 con 0x7f67ec148c40 2024-05-09T05:00:47.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.942+0000 7f67ebfff640 1 --2- 172.21.15.60:0/3150373391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67ec148c40 0x7f67ec147080 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f67e400efc0 tx=0x7f67e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:47.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.942+0000 7f67e8ff9640 1 -- 172.21.15.60:0/3150373391 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e400cbe0 con 0x7f67ec148c40 2024-05-09T05:00:47.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.942+0000 7f67e8ff9640 1 -- 172.21.15.60:0/3150373391 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67e400cd80 con 0x7f67ec148c40 2024-05-09T05:00:47.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.942+0000 7f67f1cda640 1 -- 172.21.15.60:0/3150373391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67ec161700 con 0x7f67ec148c40 2024-05-09T05:00:47.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.942+0000 7f67e8ff9640 1 -- 172.21.15.60:0/3150373391 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e4003bb0 con 0x7f67ec148c40 2024-05-09T05:00:47.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.942+0000 7f67f1cda640 1 -- 172.21.15.60:0/3150373391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67ec1f5ac0 con 0x7f67ec148c40 2024-05-09T05:00:47.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.943+0000 7f67f1cda640 1 -- 172.21.15.60:0/3150373391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67ec0d2590 con 0x7f67ec148c40 2024-05-09T05:00:47.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.944+0000 7f67e8ff9640 1 -- 172.21.15.60:0/3150373391 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f67e401b030 con 0x7f67ec148c40 2024-05-09T05:00:47.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.944+0000 7f67e8ff9640 1 --2- 172.21.15.60:0/3150373391 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f67cc0762d0 0x7f67cc078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:47.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.944+0000 7f67e8ff9640 1 -- 172.21.15.60:0/3150373391 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f67e4097c00 con 0x7f67ec148c40 2024-05-09T05:00:47.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.945+0000 7f67eb7fe640 1 --2- 172.21.15.60:0/3150373391 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f67cc0762d0 0x7f67cc078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:47.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.945+0000 7f67eb7fe640 1 --2- 172.21.15.60:0/3150373391 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f67cc0762d0 0x7f67cc078790 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f67dc034000 tx=0x7f67dc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:47.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:47.949+0000 7f67e8ff9640 1 -- 172.21.15.60:0/3150373391 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f67ec0d2590 con 0x7f67ec148c40 2024-05-09T05:00:48.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.139+0000 7f67f1cda640 1 -- 172.21.15.60:0/3150373391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f67ec0b7830 con 0x7f67ec148c40 2024-05-09T05:00:48.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.139+0000 7f67e8ff9640 1 -- 172.21.15.60:0/3150373391 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f67ec0b7830 con 0x7f67ec148c40 2024-05-09T05:00:48.139 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-05-09T05:00:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.141+0000 7f67f1cda640 1 -- 172.21.15.60:0/3150373391 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f67cc0762d0 msgr2=0x7f67cc078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.141+0000 7f67f1cda640 1 --2- 172.21.15.60:0/3150373391 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f67cc0762d0 0x7f67cc078790 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f67dc034000 tx=0x7f67dc02e000 comp rx=0 tx=0).stop 2024-05-09T05:00:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.141+0000 7f67f1cda640 1 -- 172.21.15.60:0/3150373391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67ec148c40 msgr2=0x7f67ec147080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.141+0000 7f67f1cda640 1 --2- 172.21.15.60:0/3150373391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67ec148c40 0x7f67ec147080 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f67e400efc0 tx=0x7f67e400c490 comp rx=0 tx=0).stop 2024-05-09T05:00:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.141+0000 7f67f1cda640 1 -- 172.21.15.60:0/3150373391 shutdown_connections 2024-05-09T05:00:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.141+0000 7f67f1cda640 1 --2- 172.21.15.60:0/3150373391 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f67cc0762d0 0x7f67cc078790 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.141+0000 7f67f1cda640 1 --2- 172.21.15.60:0/3150373391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67ec148c40 0x7f67ec147080 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.141+0000 7f67f1cda640 1 --2- 172.21.15.60:0/3150373391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67ec1486d0 0x7f67ec146b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.141+0000 7f67f1cda640 1 --2- 172.21.15.60:0/3150373391 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67ec0b7080 0x7f67ec143b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:48.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.141+0000 7f67f1cda640 1 -- 172.21.15.60:0/3150373391 >> 172.21.15.60:0/3150373391 conn(0x7f67ec06f6f0 msgr2=0x7f67ec0d3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:48.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.141+0000 7f67f1cda640 1 -- 172.21.15.60:0/3150373391 shutdown_connections 2024-05-09T05:00:48.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.141+0000 7f67f1cda640 1 -- 172.21.15.60:0/3150373391 wait complete. 2024-05-09T05:00:48.152 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_backtrace.TestBacktrace.test_backtrace' 2024-05-09T05:00:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.256+0000 7f39f303f640 1 Processor -- start 2024-05-09T05:00:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.256+0000 7f39f303f640 1 -- start start 2024-05-09T05:00:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.256+0000 7f39f303f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39ec06e7f0 0x7f39ec0d2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.256+0000 7f39f303f640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f39ec06fcb0 0x7f39ec0d1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.256+0000 7f39f303f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ec0b31d0 con 0x7f39ec0d2650 2024-05-09T05:00:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.257+0000 7f39f303f640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f39ec0b2f80 con 0x7f39ec06fcb0 2024-05-09T05:00:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.257+0000 7f39f303f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f39ec0b34b0 con 0x7f39ec06e7f0 2024-05-09T05:00:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.257+0000 7f39f0ddb640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39ec06e7f0 0x7f39ec0d2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.257+0000 7f39f0ddb640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39ec06e7f0 0x7f39ec0d2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.60:47546/0 (socket says 172.21.15.60:47546) 2024-05-09T05:00:48.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.257+0000 7f39f0ddb640 1 -- 172.21.15.60:0/1791235784 learned_addr learned my addr 172.21.15.60:0/1791235784 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:48.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.257+0000 7f39ebfff640 1 --2- 172.21.15.60:0/1791235784 >> v2:172.21.15.114:3300/0 conn(0x7f39ec06fcb0 0x7f39ec0d1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:48.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.257+0000 7f39f0ddb640 1 -- 172.21.15.60:0/1791235784 >> v2:172.21.15.114:3300/0 conn(0x7f39ec06fcb0 msgr2=0x7f39ec0d1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:48.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.257+0000 7f39f0ddb640 1 --2- 172.21.15.60:0/1791235784 >> v2:172.21.15.114:3300/0 conn(0x7f39ec06fcb0 0x7f39ec0d1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:48.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.257+0000 7f39f0ddb640 1 -- 172.21.15.60:0/1791235784 >> v1:172.21.15.114:6789/0 conn(0x7f39ec0d2650 legacy=0x7f39ec0d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:48.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.257+0000 7f39f0ddb640 1 -- 172.21.15.60:0/1791235784 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ec0b3790 con 0x7f39ec06e7f0 2024-05-09T05:00:48.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.258+0000 7f39f0ddb640 1 --2- 172.21.15.60:0/1791235784 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39ec06e7f0 0x7f39ec0d2b50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f39dc009870 tx=0x7f39dc02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9da64bff2af8239c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:48.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.258+0000 7f39eb7fe640 1 -- 172.21.15.60:0/1791235784 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39dc037070 con 0x7f39ec06e7f0 2024-05-09T05:00:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.258+0000 7f39eb7fe640 1 -- 172.21.15.60:0/1791235784 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39dc032440 con 0x7f39ec06e7f0 2024-05-09T05:00:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.258+0000 7f39eb7fe640 1 -- 172.21.15.60:0/1791235784 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39dc032810 con 0x7f39ec06e7f0 2024-05-09T05:00:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.259+0000 7f39f303f640 1 -- 172.21.15.60:0/1791235784 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39ec06e7f0 msgr2=0x7f39ec0d2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.259+0000 7f39f303f640 1 --2- 172.21.15.60:0/1791235784 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39ec06e7f0 0x7f39ec0d2b50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f39dc009870 tx=0x7f39dc02a270 comp rx=0 tx=0).stop 2024-05-09T05:00:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.259+0000 7f39f303f640 1 -- 172.21.15.60:0/1791235784 shutdown_connections 2024-05-09T05:00:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.259+0000 7f39f303f640 1 --2- 172.21.15.60:0/1791235784 >> v2:172.21.15.114:3300/0 conn(0x7f39ec06fcb0 0x7f39ec0d1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.259+0000 7f39f303f640 1 --2- 172.21.15.60:0/1791235784 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39ec06e7f0 0x7f39ec0d2b50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.259+0000 7f39f303f640 1 -- 172.21.15.60:0/1791235784 >> 172.21.15.60:0/1791235784 conn(0x7f39ec0993f0 msgr2=0x7f39ec099800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.259+0000 7f39f303f640 1 -- 172.21.15.60:0/1791235784 shutdown_connections 2024-05-09T05:00:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.259+0000 7f39f303f640 1 -- 172.21.15.60:0/1791235784 wait complete. 2024-05-09T05:00:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.260+0000 7f39f303f640 1 Processor -- start 2024-05-09T05:00:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.260+0000 7f39f303f640 1 -- start start 2024-05-09T05:00:48.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.261+0000 7f39f303f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39ec06e7f0 0x7f39ec15d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:48.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.261+0000 7f39f303f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39ec06fcb0 0x7f39ec15d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:48.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.261+0000 7f39f303f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39ec0d2650 0x7f39ec0b0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:48.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.261+0000 7f39f303f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec0d1e20 con 0x7f39ec06fcb0 2024-05-09T05:00:48.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.261+0000 7f39f303f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec0d1ca0 con 0x7f39ec0d2650 2024-05-09T05:00:48.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.261+0000 7f39f303f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f39ec06ebd0 con 0x7f39ec06e7f0 2024-05-09T05:00:48.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.261+0000 7f39ebfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39ec06fcb0 0x7f39ec15d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:48.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.261+0000 7f39ebfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39ec06fcb0 0x7f39ec15d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46040/0 (socket says 172.21.15.60:46040) 2024-05-09T05:00:48.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.261+0000 7f39ebfff640 1 -- 172.21.15.60:0/2925974639 learned_addr learned my addr 172.21.15.60:0/2925974639 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:48.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.261+0000 7f39f0ddb640 1 --2- 172.21.15.60:0/2925974639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39ec06e7f0 0x7f39ec15d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.261+0000 7f39f15dc640 1 --2- 172.21.15.60:0/2925974639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39ec0d2650 0x7f39ec0b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.261+0000 7f39ebfff640 1 -- 172.21.15.60:0/2925974639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39ec06e7f0 msgr2=0x7f39ec15d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.261+0000 7f39ebfff640 1 --2- 172.21.15.60:0/2925974639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39ec06e7f0 0x7f39ec15d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.261+0000 7f39ebfff640 1 -- 172.21.15.60:0/2925974639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39ec0d2650 msgr2=0x7f39ec0b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.261+0000 7f39ebfff640 1 --2- 172.21.15.60:0/2925974639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39ec0d2650 0x7f39ec0b0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.261+0000 7f39ebfff640 1 -- 172.21.15.60:0/2925974639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ec0b1750 con 0x7f39ec06fcb0 2024-05-09T05:00:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.262+0000 7f39f15dc640 1 --2- 172.21.15.60:0/2925974639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39ec0d2650 0x7f39ec0b0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.262+0000 7f39f0ddb640 1 --2- 172.21.15.60:0/2925974639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39ec06e7f0 0x7f39ec15d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.262+0000 7f39ebfff640 1 --2- 172.21.15.60:0/2925974639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39ec06fcb0 0x7f39ec15d540 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f39e000e9b0 tx=0x7f39e000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.262+0000 7f39e97fa640 1 -- 172.21.15.60:0/2925974639 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e0017070 con 0x7f39ec06fcb0 2024-05-09T05:00:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.262+0000 7f39e97fa640 1 -- 172.21.15.60:0/2925974639 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39e000c9a0 con 0x7f39ec06fcb0 2024-05-09T05:00:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.262+0000 7f39f303f640 1 -- 172.21.15.60:0/2925974639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39ec0b1a30 con 0x7f39ec06fcb0 2024-05-09T05:00:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.262+0000 7f39e97fa640 1 -- 172.21.15.60:0/2925974639 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e000cd90 con 0x7f39ec06fcb0 2024-05-09T05:00:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.262+0000 7f39f303f640 1 -- 172.21.15.60:0/2925974639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39ec0b1ff0 con 0x7f39ec06fcb0 2024-05-09T05:00:48.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.263+0000 7f39f303f640 1 -- 172.21.15.60:0/2925974639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39ec051040 con 0x7f39ec06fcb0 2024-05-09T05:00:48.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.263+0000 7f39e97fa640 1 -- 172.21.15.60:0/2925974639 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f39e001a070 con 0x7f39ec06fcb0 2024-05-09T05:00:48.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.264+0000 7f39e97fa640 1 --2- 172.21.15.60:0/2925974639 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f39d80762d0 0x7f39d8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:48.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.264+0000 7f39e97fa640 1 -- 172.21.15.60:0/2925974639 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f39e0097cf0 con 0x7f39ec06fcb0 2024-05-09T05:00:48.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.264+0000 7f39f0ddb640 1 --2- 172.21.15.60:0/2925974639 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f39d80762d0 0x7f39d8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:48.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.265+0000 7f39f0ddb640 1 --2- 172.21.15.60:0/2925974639 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f39d80762d0 0x7f39d8078790 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f39dc034000 tx=0x7f39dc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:48.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.268+0000 7f39e97fa640 1 -- 172.21.15.60:0/2925974639 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f39e009c070 con 0x7f39ec06fcb0 2024-05-09T05:00:48.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:48.538+0000 7f39f303f640 1 -- 172.21.15.60:0/2925974639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_backtrace.TestBacktrace.test_backtrace"]} v 0) -- 0x7f39ec075270 con 0x7f39ec06fcb0 2024-05-09T05:00:49.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.028+0000 7f39e97fa640 1 -- 172.21.15.60:0/2925974639 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_backtrace.TestBacktrace.test_backtrace"]}]=0 v46) ==== 138+0+0 (secure 0 0 0) 0x7f39e00613a0 con 0x7f39ec06fcb0 2024-05-09T05:00:49.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.032+0000 7f39f303f640 1 -- 172.21.15.60:0/2925974639 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f39d80762d0 msgr2=0x7f39d8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:49.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.032+0000 7f39f303f640 1 --2- 172.21.15.60:0/2925974639 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f39d80762d0 0x7f39d8078790 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f39dc034000 tx=0x7f39dc02e000 comp rx=0 tx=0).stop 2024-05-09T05:00:49.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.032+0000 7f39f303f640 1 -- 172.21.15.60:0/2925974639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39ec06fcb0 msgr2=0x7f39ec15d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:49.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.032+0000 7f39f303f640 1 --2- 172.21.15.60:0/2925974639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39ec06fcb0 0x7f39ec15d540 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f39e000e9b0 tx=0x7f39e000ee70 comp rx=0 tx=0).stop 2024-05-09T05:00:49.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.032+0000 7f39f303f640 1 -- 172.21.15.60:0/2925974639 shutdown_connections 2024-05-09T05:00:49.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.032+0000 7f39f303f640 1 --2- 172.21.15.60:0/2925974639 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f39d80762d0 0x7f39d8078790 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.032+0000 7f39f303f640 1 --2- 172.21.15.60:0/2925974639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39ec0d2650 0x7f39ec0b0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.032+0000 7f39f303f640 1 --2- 172.21.15.60:0/2925974639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39ec06fcb0 0x7f39ec15d540 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.032+0000 7f39f303f640 1 --2- 172.21.15.60:0/2925974639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f39ec06e7f0 0x7f39ec15d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.032+0000 7f39f303f640 1 -- 172.21.15.60:0/2925974639 >> 172.21.15.60:0/2925974639 conn(0x7f39ec0993f0 msgr2=0x7f39ec160820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:49.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.032+0000 7f39f303f640 1 -- 172.21.15.60:0/2925974639 shutdown_connections 2024-05-09T05:00:49.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.032+0000 7f39f303f640 1 -- 172.21.15.60:0/2925974639 wait complete. 2024-05-09T05:00:49.045 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-09T05:00:49.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.154+0000 7f0120ae3640 1 Processor -- start 2024-05-09T05:00:49.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.154+0000 7f0120ae3640 1 -- start start 2024-05-09T05:00:49.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.155+0000 7f0120ae3640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f011c0b65d0 0x7f011c0b7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:49.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.155+0000 7f0120ae3640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f011c162fa0 0x7f011c0730e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:49.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.155+0000 7f0120ae3640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f011c072c00 con 0x7f011c15f330 2024-05-09T05:00:49.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.155+0000 7f0120ae3640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f011c163540 con 0x7f011c162fa0 2024-05-09T05:00:49.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.155+0000 7f0120ae3640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f011c163820 con 0x7f011c0b65d0 2024-05-09T05:00:49.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.155+0000 7f011ad9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f011c162fa0 0x7f011c0730e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:49.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.155+0000 7f011a59c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f011c15f330 0x7f011c0b5ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56396/0 (socket says 172.21.15.60:56396) 2024-05-09T05:00:49.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.155+0000 7f011a59c640 1 -- 172.21.15.60:0/1115040404 learned_addr learned my addr 172.21.15.60:0/1115040404 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:49.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.155+0000 7f0111d9b640 1 --2- 172.21.15.60:0/1115040404 >> v2:172.21.15.114:3300/0 conn(0x7f011c0b65d0 0x7f011c0b7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:49.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.155+0000 7f011ad9d640 1 -- 172.21.15.60:0/1115040404 >> v2:172.21.15.114:3300/0 conn(0x7f011c0b65d0 msgr2=0x7f011c0b7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:49.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.155+0000 7f011ad9d640 1 --2- 172.21.15.60:0/1115040404 >> v2:172.21.15.114:3300/0 conn(0x7f011c0b65d0 0x7f011c0b7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.155+0000 7f011ad9d640 1 -- 172.21.15.60:0/1115040404 >> v1:172.21.15.60:6789/0 conn(0x7f011c15f330 legacy=0x7f011c0b5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:49.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.155+0000 7f011ad9d640 1 -- 172.21.15.60:0/1115040404 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f011c163b00 con 0x7f011c162fa0 2024-05-09T05:00:49.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.156+0000 7f011ad9d640 1 --2- 172.21.15.60:0/1115040404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f011c162fa0 0x7f011c0730e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f011c0d5dc0 tx=0x7f011402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62d210d10f557d06 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:49.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.156+0000 7f0119d9b640 1 -- 172.21.15.60:0/1115040404 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0114007740 con 0x7f011c162fa0 2024-05-09T05:00:49.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.156+0000 7f0119d9b640 1 -- 172.21.15.60:0/1115040404 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01140078e0 con 0x7f011c162fa0 2024-05-09T05:00:49.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.156+0000 7f0119d9b640 1 -- 172.21.15.60:0/1115040404 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0114007cb0 con 0x7f011c162fa0 2024-05-09T05:00:49.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.156+0000 7f0120ae3640 1 -- 172.21.15.60:0/1115040404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f011c162fa0 msgr2=0x7f011c0730e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:49.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.156+0000 7f0120ae3640 1 --2- 172.21.15.60:0/1115040404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f011c162fa0 0x7f011c0730e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f011c0d5dc0 tx=0x7f011402a680 comp rx=0 tx=0).stop 2024-05-09T05:00:49.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.156+0000 7f0120ae3640 1 -- 172.21.15.60:0/1115040404 shutdown_connections 2024-05-09T05:00:49.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.157+0000 7f0120ae3640 1 --2- 172.21.15.60:0/1115040404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f011c162fa0 0x7f011c0730e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.157+0000 7f0120ae3640 1 --2- 172.21.15.60:0/1115040404 >> v2:172.21.15.114:3300/0 conn(0x7f011c0b65d0 0x7f011c0b7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.157+0000 7f0120ae3640 1 -- 172.21.15.60:0/1115040404 >> 172.21.15.60:0/1115040404 conn(0x7f011c06ca80 msgr2=0x7f011c0d53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:49.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.157+0000 7f0120ae3640 1 -- 172.21.15.60:0/1115040404 shutdown_connections 2024-05-09T05:00:49.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.157+0000 7f0120ae3640 1 -- 172.21.15.60:0/1115040404 wait complete. 2024-05-09T05:00:49.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.158+0000 7f0120ae3640 1 Processor -- start 2024-05-09T05:00:49.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.158+0000 7f0120ae3640 1 -- start start 2024-05-09T05:00:49.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.159+0000 7f0120ae3640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f011c0b65d0 0x7f011c161c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:49.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.159+0000 7f0120ae3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f011c15f330 0x7f011c15a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:49.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.159+0000 7f0120ae3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f011c162fa0 0x7f011c14f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:49.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.159+0000 7f0120ae3640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f011c15f710 con 0x7f011c162fa0 2024-05-09T05:00:49.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.159+0000 7f0120ae3640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f011c163de0 con 0x7f011c15f330 2024-05-09T05:00:49.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.159+0000 7f0120ae3640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f011c159f20 con 0x7f011c0b65d0 2024-05-09T05:00:49.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.159+0000 7f011ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f011c162fa0 0x7f011c14f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:49.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.159+0000 7f011ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f011c162fa0 0x7f011c14f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50932/0 (socket says 172.21.15.60:50932) 2024-05-09T05:00:49.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.159+0000 7f011ad9d640 1 -- 172.21.15.60:0/3083062313 learned_addr learned my addr 172.21.15.60:0/3083062313 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:49.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.159+0000 7f0111d9b640 1 --2- 172.21.15.60:0/3083062313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f011c15f330 0x7f011c15a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:49.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.159+0000 7f011a59c640 1 --2- 172.21.15.60:0/3083062313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f011c0b65d0 0x7f011c161c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:49.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.159+0000 7f011ad9d640 1 -- 172.21.15.60:0/3083062313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f011c0b65d0 msgr2=0x7f011c161c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.159+0000 7f011ad9d640 1 --2- 172.21.15.60:0/3083062313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f011c0b65d0 0x7f011c161c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.159+0000 7f011ad9d640 1 -- 172.21.15.60:0/3083062313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f011c15f330 msgr2=0x7f011c15a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.159+0000 7f011ad9d640 1 --2- 172.21.15.60:0/3083062313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f011c15f330 0x7f011c15a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.159+0000 7f011ad9d640 1 -- 172.21.15.60:0/3083062313 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f011c15af30 con 0x7f011c162fa0 2024-05-09T05:00:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.159+0000 7f011ad9d640 1 --2- 172.21.15.60:0/3083062313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f011c162fa0 0x7f011c14f220 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f011402ac70 tx=0x7f0114004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.160+0000 7f01137fe640 1 -- 172.21.15.60:0/3083062313 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f011403f070 con 0x7f011c162fa0 2024-05-09T05:00:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.160+0000 7f01137fe640 1 -- 172.21.15.60:0/3083062313 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0114031bc0 con 0x7f011c162fa0 2024-05-09T05:00:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.160+0000 7f0120ae3640 1 -- 172.21.15.60:0/3083062313 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f011c1595f0 con 0x7f011c162fa0 2024-05-09T05:00:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.160+0000 7f01137fe640 1 -- 172.21.15.60:0/3083062313 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0114044460 con 0x7f011c162fa0 2024-05-09T05:00:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.160+0000 7f0120ae3640 1 -- 172.21.15.60:0/3083062313 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f011c159bb0 con 0x7f011c162fa0 2024-05-09T05:00:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.161+0000 7f0120ae3640 1 -- 172.21.15.60:0/3083062313 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f011c15a390 con 0x7f011c162fa0 2024-05-09T05:00:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.162+0000 7f01137fe640 1 -- 172.21.15.60:0/3083062313 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0114007790 con 0x7f011c162fa0 2024-05-09T05:00:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.162+0000 7f01137fe640 1 --2- 172.21.15.60:0/3083062313 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f00fc0762d0 0x7f00fc078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.162+0000 7f01137fe640 1 -- 172.21.15.60:0/3083062313 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f01140b9090 con 0x7f011c162fa0 2024-05-09T05:00:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.162+0000 7f011a59c640 1 --2- 172.21.15.60:0/3083062313 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f00fc0762d0 0x7f00fc078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.163+0000 7f011a59c640 1 --2- 172.21.15.60:0/3083062313 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f00fc0762d0 0x7f00fc078790 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f010c007fd0 tx=0x7f010c007ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:49.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.166+0000 7f01137fe640 1 -- 172.21.15.60:0/3083062313 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f011c15a390 con 0x7f011c162fa0 2024-05-09T05:00:49.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.360+0000 7f0120ae3640 1 -- 172.21.15.60:0/3083062313 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f011c051040 con 0x7f011c162fa0 2024-05-09T05:00:49.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.363+0000 7f01137fe640 1 -- 172.21.15.60:0/3083062313 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f0114082060 con 0x7f011c162fa0 2024-05-09T05:00:49.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.365+0000 7f0120ae3640 1 -- 172.21.15.60:0/3083062313 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f00fc0762d0 msgr2=0x7f00fc078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:49.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.365+0000 7f0120ae3640 1 --2- 172.21.15.60:0/3083062313 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f00fc0762d0 0x7f00fc078790 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f010c007fd0 tx=0x7f010c007ee0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.365+0000 7f0120ae3640 1 -- 172.21.15.60:0/3083062313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f011c162fa0 msgr2=0x7f011c14f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:49.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.365+0000 7f0120ae3640 1 --2- 172.21.15.60:0/3083062313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f011c162fa0 0x7f011c14f220 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f011402ac70 tx=0x7f0114004a40 comp rx=0 tx=0).stop 2024-05-09T05:00:49.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.365+0000 7f0120ae3640 1 -- 172.21.15.60:0/3083062313 shutdown_connections 2024-05-09T05:00:49.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.365+0000 7f0120ae3640 1 --2- 172.21.15.60:0/3083062313 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f00fc0762d0 0x7f00fc078790 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.366 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.365+0000 7f0120ae3640 1 --2- 172.21.15.60:0/3083062313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f011c162fa0 0x7f011c14f220 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.365+0000 7f0120ae3640 1 --2- 172.21.15.60:0/3083062313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f011c15f330 0x7f011c15a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.365+0000 7f0120ae3640 1 --2- 172.21.15.60:0/3083062313 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f011c0b65d0 0x7f011c161c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.365+0000 7f0120ae3640 1 -- 172.21.15.60:0/3083062313 >> 172.21.15.60:0/3083062313 conn(0x7f011c06ca80 msgr2=0x7f011c152080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:49.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.365+0000 7f0120ae3640 1 -- 172.21.15.60:0/3083062313 shutdown_connections 2024-05-09T05:00:49.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.365+0000 7f0120ae3640 1 -- 172.21.15.60:0/3083062313 wait complete. 2024-05-09T05:00:49.377 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T05:00:49.377 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T05:00:49.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T05:00:49.463 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T05:00:49.463 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:00:49.463 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:00:49.480 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:00:49.480 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:00:49.537 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:00:49.537 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:00:49.594 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T05:00:49.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.739+0000 7fd27e0c8640 1 Processor -- start 2024-05-09T05:00:49.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.739+0000 7fd27e0c8640 1 -- start start 2024-05-09T05:00:49.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.740+0000 7fd27e0c8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd27806f9f0 0x7fd27806ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:49.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.740+0000 7fd27e0c8640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd27806ff60 con 0x7fd2781606f0 2024-05-09T05:00:49.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.740+0000 7fd27e0c8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd278070240 con 0x7fd27815f300 2024-05-09T05:00:49.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.740+0000 7fd27e0c8640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fd278070520 con 0x7fd27806f9f0 2024-05-09T05:00:49.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.740+0000 7fd277fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd27806f9f0 0x7fd27806ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:49.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.740+0000 7fd277fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd27806f9f0 0x7fd27806ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50938/0 (socket says 172.21.15.60:50938) 2024-05-09T05:00:49.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.740+0000 7fd277fff640 1 -- 172.21.15.60:0/3007250346 learned_addr learned my addr 172.21.15.60:0/3007250346 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:49.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.740+0000 7fd277fff640 1 -- 172.21.15.60:0/3007250346 >> v1:172.21.15.114:6789/0 conn(0x7fd27815f300 legacy=0x7fd2781639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:49.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.740+0000 7fd277fff640 1 -- 172.21.15.60:0/3007250346 >> v1:172.21.15.60:6789/0 conn(0x7fd2781606f0 legacy=0x7fd2780d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:49.741 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.740+0000 7fd277fff640 1 -- 172.21.15.60:0/3007250346 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd27806e740 con 0x7fd27806f9f0 2024-05-09T05:00:49.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.741+0000 7fd277fff640 1 --2- 172.21.15.60:0/3007250346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd27806f9f0 0x7fd27806ef40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd270009e80 tx=0x7fd27002a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0c88a0018999cb5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:49.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.741+0000 7fd2767fc640 1 -- 172.21.15.60:0/3007250346 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd270036070 con 0x7fd27806f9f0 2024-05-09T05:00:49.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.741+0000 7fd2767fc640 1 -- 172.21.15.60:0/3007250346 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd270031600 con 0x7fd27806f9f0 2024-05-09T05:00:49.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.741+0000 7fd2767fc640 1 -- 172.21.15.60:0/3007250346 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2700319d0 con 0x7fd27806f9f0 2024-05-09T05:00:49.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.742+0000 7fd27e0c8640 1 -- 172.21.15.60:0/3007250346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd27806f9f0 msgr2=0x7fd27806ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:49.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.742+0000 7fd27e0c8640 1 --2- 172.21.15.60:0/3007250346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd27806f9f0 0x7fd27806ef40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd270009e80 tx=0x7fd27002a750 comp rx=0 tx=0).stop 2024-05-09T05:00:49.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.742+0000 7fd27e0c8640 1 -- 172.21.15.60:0/3007250346 shutdown_connections 2024-05-09T05:00:49.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.742+0000 7fd27e0c8640 1 --2- 172.21.15.60:0/3007250346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd27806f9f0 0x7fd27806ef40 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.742+0000 7fd27e0c8640 1 -- 172.21.15.60:0/3007250346 >> 172.21.15.60:0/3007250346 conn(0x7fd27809b850 msgr2=0x7fd27809bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:49.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.742+0000 7fd27e0c8640 1 -- 172.21.15.60:0/3007250346 shutdown_connections 2024-05-09T05:00:49.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.742+0000 7fd27e0c8640 1 -- 172.21.15.60:0/3007250346 wait complete. 2024-05-09T05:00:49.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.743+0000 7fd27e0c8640 1 Processor -- start 2024-05-09T05:00:49.744 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.743+0000 7fd27e0c8640 1 -- start start 2024-05-09T05:00:49.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.744+0000 7fd27e0c8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd27806f9f0 0x7fd27809def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:49.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.744+0000 7fd27e0c8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd27815f300 0x7fd2780ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:49.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.744+0000 7fd27e0c8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2781606f0 0x7fd2780992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:49.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.744+0000 7fd27e0c8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd2780d3c80 con 0x7fd27806f9f0 2024-05-09T05:00:49.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.744+0000 7fd27e0c8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd2780d3b00 con 0x7fd2781606f0 2024-05-09T05:00:49.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.744+0000 7fd27e0c8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd27806f480 con 0x7fd27815f300 2024-05-09T05:00:49.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.744+0000 7fd2777fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd27806f9f0 0x7fd27809def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:49.746 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.744+0000 7fd2777fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd27806f9f0 0x7fd27809def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50954/0 (socket says 172.21.15.60:50954) 2024-05-09T05:00:49.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.744+0000 7fd2777fe640 1 -- 172.21.15.60:0/2837293831 learned_addr learned my addr 172.21.15.60:0/2837293831 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:49.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.744+0000 7fd277fff640 1 --2- 172.21.15.60:0/2837293831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2781606f0 0x7fd2780992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:49.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.744+0000 7fd2777fe640 1 -- 172.21.15.60:0/2837293831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd27815f300 msgr2=0x7fd2780ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:49.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.744+0000 7fd276ffd640 1 --2- 172.21.15.60:0/2837293831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd27815f300 0x7fd2780ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:49.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.744+0000 7fd2777fe640 1 --2- 172.21.15.60:0/2837293831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd27815f300 0x7fd2780ac2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.744+0000 7fd2777fe640 1 -- 172.21.15.60:0/2837293831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2781606f0 msgr2=0x7fd2780992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:49.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.744+0000 7fd2777fe640 1 --2- 172.21.15.60:0/2837293831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2781606f0 0x7fd2780992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.744+0000 7fd2777fe640 1 -- 172.21.15.60:0/2837293831 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd278163f00 con 0x7fd27806f9f0 2024-05-09T05:00:49.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.744+0000 7fd277fff640 1 --2- 172.21.15.60:0/2837293831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2781606f0 0x7fd2780992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:49.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.744+0000 7fd276ffd640 1 --2- 172.21.15.60:0/2837293831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd27815f300 0x7fd2780ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:49.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.745+0000 7fd2777fe640 1 --2- 172.21.15.60:0/2837293831 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd27806f9f0 0x7fd27809def0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd26800c430 tx=0x7fd26800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:49.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.745+0000 7fd25ffff640 1 -- 172.21.15.60:0/2837293831 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd268011790 con 0x7fd27806f9f0 2024-05-09T05:00:49.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.745+0000 7fd25ffff640 1 -- 172.21.15.60:0/2837293831 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd268017070 con 0x7fd27806f9f0 2024-05-09T05:00:49.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.745+0000 7fd27e0c8640 1 -- 172.21.15.60:0/2837293831 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd27814d970 con 0x7fd27806f9f0 2024-05-09T05:00:49.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.745+0000 7fd25ffff640 1 -- 172.21.15.60:0/2837293831 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd268011bb0 con 0x7fd27806f9f0 2024-05-09T05:00:49.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.745+0000 7fd27e0c8640 1 -- 172.21.15.60:0/2837293831 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd27809c840 con 0x7fd27806f9f0 2024-05-09T05:00:49.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.746+0000 7fd27e0c8640 1 -- 172.21.15.60:0/2837293831 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd278051040 con 0x7fd27806f9f0 2024-05-09T05:00:49.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.746+0000 7fd25ffff640 1 -- 172.21.15.60:0/2837293831 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd26801c460 con 0x7fd27806f9f0 2024-05-09T05:00:49.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.747+0000 7fd25ffff640 1 --2- 172.21.15.60:0/2837293831 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd2640762d0 0x7fd264078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:49.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.747+0000 7fd25ffff640 1 -- 172.21.15.60:0/2837293831 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd268097730 con 0x7fd27806f9f0 2024-05-09T05:00:49.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.747+0000 7fd276ffd640 1 --2- 172.21.15.60:0/2837293831 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd2640762d0 0x7fd264078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:49.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.748+0000 7fd276ffd640 1 --2- 172.21.15.60:0/2837293831 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd2640762d0 0x7fd264078790 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd27809c260 tx=0x7fd26c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:49.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.751+0000 7fd25ffff640 1 -- 172.21.15.60:0/2837293831 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd268060de0 con 0x7fd27806f9f0 2024-05-09T05:00:49.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.959+0000 7fd27e0c8640 1 -- 172.21.15.60:0/2837293831 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd27815d1e0 con 0x7fd27806f9f0 2024-05-09T05:00:49.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.959+0000 7fd25ffff640 1 -- 172.21.15.60:0/2837293831 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7fd2680606e0 con 0x7fd27806f9f0 2024-05-09T05:00:49.960 INFO:teuthology.orchestra.run.smithi060.stderr:listed 0 entries 2024-05-09T05:00:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.961+0000 7fd27e0c8640 1 -- 172.21.15.60:0/2837293831 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd2640762d0 msgr2=0x7fd264078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.961+0000 7fd27e0c8640 1 --2- 172.21.15.60:0/2837293831 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd2640762d0 0x7fd264078790 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd27809c260 tx=0x7fd26c009040 comp rx=0 tx=0).stop 2024-05-09T05:00:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.961+0000 7fd27e0c8640 1 -- 172.21.15.60:0/2837293831 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd27806f9f0 msgr2=0x7fd27809def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.961+0000 7fd27e0c8640 1 --2- 172.21.15.60:0/2837293831 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd27806f9f0 0x7fd27809def0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd26800c430 tx=0x7fd26800c8f0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.961+0000 7fd27e0c8640 1 -- 172.21.15.60:0/2837293831 shutdown_connections 2024-05-09T05:00:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.961+0000 7fd27e0c8640 1 --2- 172.21.15.60:0/2837293831 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd2640762d0 0x7fd264078790 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.961+0000 7fd27e0c8640 1 --2- 172.21.15.60:0/2837293831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2781606f0 0x7fd2780992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.961+0000 7fd27e0c8640 1 --2- 172.21.15.60:0/2837293831 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd27815f300 0x7fd2780ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.961+0000 7fd27e0c8640 1 --2- 172.21.15.60:0/2837293831 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd27806f9f0 0x7fd27809def0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.961+0000 7fd27e0c8640 1 -- 172.21.15.60:0/2837293831 >> 172.21.15.60:0/2837293831 conn(0x7fd27809b850 msgr2=0x7fd27815d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.962+0000 7fd27e0c8640 1 -- 172.21.15.60:0/2837293831 shutdown_connections 2024-05-09T05:00:49.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:49.962+0000 7fd27e0c8640 1 -- 172.21.15.60:0/2837293831 wait complete. 2024-05-09T05:00:49.973 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-09T05:00:49.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:49.973 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-05-09T05:00:50.000 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2024-05-09T05:00:50.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:50.001 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T05:00:50.057 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T05:00:50.057 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:00:50.076 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T05:00:50.077 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:00:50.135 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T05:00:50.135 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:00:50.194 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T05:00:50.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.299+0000 7f02bbb5f640 1 Processor -- start 2024-05-09T05:00:50.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.299+0000 7f02bbb5f640 1 -- start start 2024-05-09T05:00:50.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.300+0000 7f02bbb5f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f02b4072130 0x7f02b40725b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:50.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.300+0000 7f02bbb5f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02b40adbb0 0x7f02b40ae040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:50.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.300+0000 7f02bbb5f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b4159d10 con 0x7f02b4159930 2024-05-09T05:00:50.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.300+0000 7f02bbb5f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f02b406f8e0 con 0x7f02b4072130 2024-05-09T05:00:50.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.300+0000 7f02bbb5f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f02b406fbc0 con 0x7f02b40adbb0 2024-05-09T05:00:50.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.300+0000 7f02b90fa640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f02b4072130 0x7f02b40725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:50.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.300+0000 7f02b98fb640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f02b4159930 0x7f02b41601b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56416/0 (socket says 172.21.15.60:56416) 2024-05-09T05:00:50.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.300+0000 7f02b90fa640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f02b4072130 0x7f02b40725b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50964/0 (socket says 172.21.15.60:50964) 2024-05-09T05:00:50.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.300+0000 7f02b98fb640 1 -- 172.21.15.60:0/2943271267 learned_addr learned my addr 172.21.15.60:0/2943271267 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:50.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.300+0000 7f02ba0fc640 1 --2- 172.21.15.60:0/2943271267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02b40adbb0 0x7f02b40ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:50.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.300+0000 7f02b90fa640 1 -- 172.21.15.60:0/2943271267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02b40adbb0 msgr2=0x7f02b40ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:50.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.300+0000 7f02b90fa640 1 --2- 172.21.15.60:0/2943271267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02b40adbb0 0x7f02b40ae040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:50.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.300+0000 7f02b90fa640 1 -- 172.21.15.60:0/2943271267 >> v1:172.21.15.60:6789/0 conn(0x7f02b4159930 legacy=0x7f02b41601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:50.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.300+0000 7f02b88f9640 1 -- 172.21.15.60:0/2943271267 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 7365396 0 0) 0x7f02b0001110 con 0x7f02b4159930 2024-05-09T05:00:50.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.300+0000 7f02b90fa640 1 -- 172.21.15.60:0/2943271267 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b406fea0 con 0x7f02b4072130 2024-05-09T05:00:50.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.300+0000 7f02ba0fc640 1 --2- 172.21.15.60:0/2943271267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02b40adbb0 0x7f02b40ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:50.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.301+0000 7f02b90fa640 1 --2- 172.21.15.60:0/2943271267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b4072130 0x7f02b40725b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f02a40097a0 tx=0x7f02a402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34cc504dd027da01 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:50.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.301+0000 7f02b88f9640 1 -- 172.21.15.60:0/2943271267 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4037070 con 0x7f02b4072130 2024-05-09T05:00:50.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.301+0000 7f02b88f9640 1 -- 172.21.15.60:0/2943271267 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02a4032440 con 0x7f02b4072130 2024-05-09T05:00:50.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.301+0000 7f02b88f9640 1 -- 172.21.15.60:0/2943271267 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4032810 con 0x7f02b4072130 2024-05-09T05:00:50.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.302+0000 7f02bbb5f640 1 -- 172.21.15.60:0/2943271267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b4072130 msgr2=0x7f02b40725b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:50.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.302+0000 7f02bbb5f640 1 --2- 172.21.15.60:0/2943271267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b4072130 0x7f02b40725b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f02a40097a0 tx=0x7f02a402a180 comp rx=0 tx=0).stop 2024-05-09T05:00:50.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.302+0000 7f02bbb5f640 1 -- 172.21.15.60:0/2943271267 shutdown_connections 2024-05-09T05:00:50.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.302+0000 7f02bbb5f640 1 --2- 172.21.15.60:0/2943271267 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02b40adbb0 0x7f02b40ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:50.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.302+0000 7f02bbb5f640 1 --2- 172.21.15.60:0/2943271267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b4072130 0x7f02b40725b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:50.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.302+0000 7f02bbb5f640 1 -- 172.21.15.60:0/2943271267 >> 172.21.15.60:0/2943271267 conn(0x7f02b40b4c30 msgr2=0x7f02b4073240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:50.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.302+0000 7f02bbb5f640 1 -- 172.21.15.60:0/2943271267 shutdown_connections 2024-05-09T05:00:50.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.302+0000 7f02bbb5f640 1 -- 172.21.15.60:0/2943271267 wait complete. 2024-05-09T05:00:50.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.303+0000 7f02bbb5f640 1 Processor -- start 2024-05-09T05:00:50.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.303+0000 7f02bbb5f640 1 -- start start 2024-05-09T05:00:50.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.304+0000 7f02bbb5f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02b4072130 0x7f02b4157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:50.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.304+0000 7f02bbb5f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02b40adbb0 0x7f02b415c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:50.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.304+0000 7f02bbb5f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b4159930 0x7f02b4162c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:50.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.304+0000 7f02bbb5f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f02b415fae0 con 0x7f02b4159930 2024-05-09T05:00:50.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.304+0000 7f02bbb5f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f02b4070460 con 0x7f02b40adbb0 2024-05-09T05:00:50.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.304+0000 7f02bbb5f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f02b41eaaa0 con 0x7f02b4072130 2024-05-09T05:00:50.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.304+0000 7f02ba0fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b4159930 0x7f02b4162c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:50.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.304+0000 7f02ba0fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b4159930 0x7f02b4162c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50972/0 (socket says 172.21.15.60:50972) 2024-05-09T05:00:50.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.304+0000 7f02ba0fc640 1 -- 172.21.15.60:0/3168154148 learned_addr learned my addr 172.21.15.60:0/3168154148 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:50.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.304+0000 7f02b98fb640 1 --2- 172.21.15.60:0/3168154148 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02b4072130 0x7f02b4157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:50.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.304+0000 7f02b90fa640 1 --2- 172.21.15.60:0/3168154148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02b40adbb0 0x7f02b415c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:50.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.304+0000 7f02ba0fc640 1 -- 172.21.15.60:0/3168154148 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02b4072130 msgr2=0x7f02b4157480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.304+0000 7f02ba0fc640 1 --2- 172.21.15.60:0/3168154148 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02b4072130 0x7f02b4157480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.304+0000 7f02ba0fc640 1 -- 172.21.15.60:0/3168154148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02b40adbb0 msgr2=0x7f02b415c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.304+0000 7f02ba0fc640 1 --2- 172.21.15.60:0/3168154148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02b40adbb0 0x7f02b415c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.304+0000 7f02ba0fc640 1 -- 172.21.15.60:0/3168154148 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b41e9a00 con 0x7f02b4159930 2024-05-09T05:00:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.304+0000 7f02b90fa640 1 --2- 172.21.15.60:0/3168154148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02b40adbb0 0x7f02b415c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.304+0000 7f02ba0fc640 1 --2- 172.21.15.60:0/3168154148 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b4159930 0x7f02b4162c20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f02ac00cef0 tx=0x7f02ac00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.305+0000 7f02aa7fc640 1 -- 172.21.15.60:0/3168154148 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac01a070 con 0x7f02b4159930 2024-05-09T05:00:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.305+0000 7f02aa7fc640 1 -- 172.21.15.60:0/3168154148 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02ac00d860 con 0x7f02b4159930 2024-05-09T05:00:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.305+0000 7f02bbb5f640 1 -- 172.21.15.60:0/3168154148 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02b41e9ce0 con 0x7f02b4159930 2024-05-09T05:00:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.305+0000 7f02aa7fc640 1 -- 172.21.15.60:0/3168154148 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac00dc50 con 0x7f02b4159930 2024-05-09T05:00:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.305+0000 7f02bbb5f640 1 -- 172.21.15.60:0/3168154148 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02b41ea2a0 con 0x7f02b4159930 2024-05-09T05:00:50.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.306+0000 7f02bbb5f640 1 -- 172.21.15.60:0/3168154148 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02b415c200 con 0x7f02b4159930 2024-05-09T05:00:50.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.306+0000 7f02aa7fc640 1 -- 172.21.15.60:0/3168154148 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f02ac01d070 con 0x7f02b4159930 2024-05-09T05:00:50.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.307+0000 7f02aa7fc640 1 --2- 172.21.15.60:0/3168154148 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f02980762d0 0x7f0298078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:50.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.307+0000 7f02aa7fc640 1 -- 172.21.15.60:0/3168154148 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f02ac0981b0 con 0x7f02b4159930 2024-05-09T05:00:50.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.307+0000 7f02b98fb640 1 --2- 172.21.15.60:0/3168154148 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f02980762d0 0x7f0298078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:50.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.308+0000 7f02b98fb640 1 --2- 172.21.15.60:0/3168154148 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f02980762d0 0x7f0298078790 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f02b0000f60 tx=0x7f02b0007c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:50.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.311+0000 7f02aa7fc640 1 -- 172.21.15.60:0/3168154148 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f02ac061880 con 0x7f02b4159930 2024-05-09T05:00:50.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.513+0000 7f02bbb5f640 1 -- 172.21.15.60:0/3168154148 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f02b406c4b0 con 0x7f02b4159930 2024-05-09T05:00:50.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.513+0000 7f02aa7fc640 1 -- 172.21.15.60:0/3168154148 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f02ac061180 con 0x7f02b4159930 2024-05-09T05:00:50.514 INFO:teuthology.orchestra.run.smithi060.stderr:listed 0 entries 2024-05-09T05:00:50.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.515+0000 7f02bbb5f640 1 -- 172.21.15.60:0/3168154148 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f02980762d0 msgr2=0x7f0298078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:50.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.515+0000 7f02bbb5f640 1 --2- 172.21.15.60:0/3168154148 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f02980762d0 0x7f0298078790 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f02b0000f60 tx=0x7f02b0007c00 comp rx=0 tx=0).stop 2024-05-09T05:00:50.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.515+0000 7f02bbb5f640 1 -- 172.21.15.60:0/3168154148 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b4159930 msgr2=0x7f02b4162c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:50.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.515+0000 7f02bbb5f640 1 --2- 172.21.15.60:0/3168154148 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b4159930 0x7f02b4162c20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f02ac00cef0 tx=0x7f02ac00a710 comp rx=0 tx=0).stop 2024-05-09T05:00:50.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.515+0000 7f02bbb5f640 1 -- 172.21.15.60:0/3168154148 shutdown_connections 2024-05-09T05:00:50.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.515+0000 7f02bbb5f640 1 --2- 172.21.15.60:0/3168154148 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f02980762d0 0x7f0298078790 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:50.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.515+0000 7f02bbb5f640 1 --2- 172.21.15.60:0/3168154148 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b4159930 0x7f02b4162c20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:50.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.515+0000 7f02bbb5f640 1 --2- 172.21.15.60:0/3168154148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02b40adbb0 0x7f02b415c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:50.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.515+0000 7f02bbb5f640 1 --2- 172.21.15.60:0/3168154148 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f02b4072130 0x7f02b4157480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:50.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.515+0000 7f02bbb5f640 1 -- 172.21.15.60:0/3168154148 >> 172.21.15.60:0/3168154148 conn(0x7f02b40b4c30 msgr2=0x7f02b41609f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:50.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.515+0000 7f02bbb5f640 1 -- 172.21.15.60:0/3168154148 shutdown_connections 2024-05-09T05:00:50.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.515+0000 7f02bbb5f640 1 -- 172.21.15.60:0/3168154148 wait complete. 2024-05-09T05:00:50.527 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-05-09T05:00:50.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:50.527 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-05-09T05:00:50.558 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T05:00:50.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:50.558 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-05-09T05:00:50.622 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:00:50.622 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:00:50.639 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:00:50.640 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:00:50.696 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:00:50.696 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:00:50.753 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T05:00:50.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.897+0000 7f4e6150a640 1 Processor -- start 2024-05-09T05:00:50.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.898+0000 7f4e6150a640 1 -- start start 2024-05-09T05:00:50.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.898+0000 7f4e6150a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4e5c15d3b0 0x7f4e5c15d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:50.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.898+0000 7f4e6150a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e5c074750 0x7f4e5c073a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:50.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.898+0000 7f4e6150a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4e5c097de0 0x7f4e5c073f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:50.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.898+0000 7f4e6150a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f4e5c06e7b0 con 0x7f4e5c15d3b0 2024-05-09T05:00:50.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.898+0000 7f4e6150a640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f4e5c06f610 con 0x7f4e5c097de0 2024-05-09T05:00:50.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.898+0000 7f4e6150a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4e5c06ea00 con 0x7f4e5c074750 2024-05-09T05:00:50.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.898+0000 7f4e5affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4e5c15d3b0 0x7f4e5c15d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:50.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.898+0000 7f4e5affd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4e5c15d3b0 0x7f4e5c15d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50980/0 (socket says 172.21.15.60:50980) 2024-05-09T05:00:50.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.899+0000 7f4e5affd640 1 -- 172.21.15.60:0/612307400 learned_addr learned my addr 172.21.15.60:0/612307400 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:50.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.899+0000 7f4e5b7fe640 1 --2- 172.21.15.60:0/612307400 >> v2:172.21.15.114:3300/0 conn(0x7f4e5c097de0 0x7f4e5c073f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:50.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.899+0000 7f4e5a7fc640 1 --2- 172.21.15.60:0/612307400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e5c074750 0x7f4e5c073a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:50.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.899+0000 7f4e5affd640 1 -- 172.21.15.60:0/612307400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e5c074750 msgr2=0x7f4e5c073a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:50.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.899+0000 7f4e5affd640 1 --2- 172.21.15.60:0/612307400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e5c074750 0x7f4e5c073a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:50.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.899+0000 7f4e5affd640 1 -- 172.21.15.60:0/612307400 >> v2:172.21.15.114:3300/0 conn(0x7f4e5c097de0 msgr2=0x7f4e5c073f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:50.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.899+0000 7f4e5affd640 1 --2- 172.21.15.60:0/612307400 >> v2:172.21.15.114:3300/0 conn(0x7f4e5c097de0 0x7f4e5c073f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:50.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.899+0000 7f4e5affd640 1 -- 172.21.15.60:0/612307400 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e5c06ece0 con 0x7f4e5c15d3b0 2024-05-09T05:00:50.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.899+0000 7f4e5affd640 1 --2- 172.21.15.60:0/612307400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e5c15d3b0 0x7f4e5c15d790 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c0b2270 tx=0x7f4e5402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c7d4ffcdc34c77 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:50.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.900+0000 7f4e59ffb640 1 -- 172.21.15.60:0/612307400 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e54037070 con 0x7f4e5c15d3b0 2024-05-09T05:00:50.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.900+0000 7f4e59ffb640 1 -- 172.21.15.60:0/612307400 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e54032440 con 0x7f4e5c15d3b0 2024-05-09T05:00:50.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.900+0000 7f4e59ffb640 1 -- 172.21.15.60:0/612307400 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e54032810 con 0x7f4e5c15d3b0 2024-05-09T05:00:50.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.900+0000 7f4e6150a640 1 -- 172.21.15.60:0/612307400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e5c15d3b0 msgr2=0x7f4e5c15d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:50.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.900+0000 7f4e6150a640 1 --2- 172.21.15.60:0/612307400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e5c15d3b0 0x7f4e5c15d790 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c0b2270 tx=0x7f4e5402a140 comp rx=0 tx=0).stop 2024-05-09T05:00:50.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.900+0000 7f4e6150a640 1 -- 172.21.15.60:0/612307400 shutdown_connections 2024-05-09T05:00:50.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.900+0000 7f4e6150a640 1 --2- 172.21.15.60:0/612307400 >> v2:172.21.15.114:3300/0 conn(0x7f4e5c097de0 0x7f4e5c073f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:50.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.900+0000 7f4e6150a640 1 --2- 172.21.15.60:0/612307400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e5c074750 0x7f4e5c073a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:50.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.900+0000 7f4e6150a640 1 --2- 172.21.15.60:0/612307400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e5c15d3b0 0x7f4e5c15d790 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:50.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.900+0000 7f4e6150a640 1 -- 172.21.15.60:0/612307400 >> 172.21.15.60:0/612307400 conn(0x7f4e5c0972f0 msgr2=0x7f4e5c0976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:50.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.900+0000 7f4e6150a640 1 -- 172.21.15.60:0/612307400 shutdown_connections 2024-05-09T05:00:50.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.900+0000 7f4e6150a640 1 -- 172.21.15.60:0/612307400 wait complete. 2024-05-09T05:00:50.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.902+0000 7f4e6150a640 1 Processor -- start 2024-05-09T05:00:50.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.902+0000 7f4e6150a640 1 -- start start 2024-05-09T05:00:50.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.902+0000 7f4e6150a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e5c074750 0x7f4e5c15a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:50.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.902+0000 7f4e6150a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e5c097de0 0x7f4e5c0ac8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:50.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.902+0000 7f4e6150a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e5c15d3b0 0x7f4e5c1f4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:50.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.902+0000 7f4e6150a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5c0b4540 con 0x7f4e5c15d3b0 2024-05-09T05:00:50.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.902+0000 7f4e6150a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5c06f2a0 con 0x7f4e5c097de0 2024-05-09T05:00:50.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.902+0000 7f4e6150a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4e5c15dcd0 con 0x7f4e5c074750 2024-05-09T05:00:50.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.902+0000 7f4e5b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e5c15d3b0 0x7f4e5c1f4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:50.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.903+0000 7f4e5b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e5c15d3b0 0x7f4e5c1f4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50986/0 (socket says 172.21.15.60:50986) 2024-05-09T05:00:50.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.903+0000 7f4e5b7fe640 1 -- 172.21.15.60:0/3417173116 learned_addr learned my addr 172.21.15.60:0/3417173116 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:50.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.903+0000 7f4e5a7fc640 1 --2- 172.21.15.60:0/3417173116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e5c097de0 0x7f4e5c0ac8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:50.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.903+0000 7f4e5affd640 1 --2- 172.21.15.60:0/3417173116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e5c074750 0x7f4e5c15a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:50.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.903+0000 7f4e5b7fe640 1 -- 172.21.15.60:0/3417173116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e5c074750 msgr2=0x7f4e5c15a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:50.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.903+0000 7f4e5b7fe640 1 --2- 172.21.15.60:0/3417173116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e5c074750 0x7f4e5c15a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:50.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.903+0000 7f4e5b7fe640 1 -- 172.21.15.60:0/3417173116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e5c097de0 msgr2=0x7f4e5c0ac8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:50.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.903+0000 7f4e5b7fe640 1 --2- 172.21.15.60:0/3417173116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e5c097de0 0x7f4e5c0ac8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:50.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.903+0000 7f4e5b7fe640 1 -- 172.21.15.60:0/3417173116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e5c0abd50 con 0x7f4e5c15d3b0 2024-05-09T05:00:50.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.903+0000 7f4e5a7fc640 1 --2- 172.21.15.60:0/3417173116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e5c097de0 0x7f4e5c0ac8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:50.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.903+0000 7f4e5b7fe640 1 --2- 172.21.15.60:0/3417173116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e5c15d3b0 0x7f4e5c1f4d00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c007d90 tx=0x7f4e4c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:50.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.903+0000 7f4e537fe640 1 -- 172.21.15.60:0/3417173116 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c014420 con 0x7f4e5c15d3b0 2024-05-09T05:00:50.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.903+0000 7f4e537fe640 1 -- 172.21.15.60:0/3417173116 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e4c019070 con 0x7f4e5c15d3b0 2024-05-09T05:00:50.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.903+0000 7f4e537fe640 1 -- 172.21.15.60:0/3417173116 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c014820 con 0x7f4e5c15d3b0 2024-05-09T05:00:50.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.903+0000 7f4e6150a640 1 -- 172.21.15.60:0/3417173116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e5c1517a0 con 0x7f4e5c15d3b0 2024-05-09T05:00:50.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.903+0000 7f4e6150a640 1 -- 172.21.15.60:0/3417173116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e5c151d60 con 0x7f4e5c15d3b0 2024-05-09T05:00:50.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.904+0000 7f4e6150a640 1 -- 172.21.15.60:0/3417173116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e5c151f60 con 0x7f4e5c15d3b0 2024-05-09T05:00:50.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.905+0000 7f4e537fe640 1 -- 172.21.15.60:0/3417173116 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4e4c0149c0 con 0x7f4e5c15d3b0 2024-05-09T05:00:50.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.905+0000 7f4e537fe640 1 --2- 172.21.15.60:0/3417173116 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4e3c0762d0 0x7f4e3c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:50.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.905+0000 7f4e537fe640 1 -- 172.21.15.60:0/3417173116 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4e4c097fc0 con 0x7f4e5c15d3b0 2024-05-09T05:00:50.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.905+0000 7f4e5affd640 1 --2- 172.21.15.60:0/3417173116 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4e3c0762d0 0x7f4e3c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:50.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.906+0000 7f4e5affd640 1 --2- 172.21.15.60:0/3417173116 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4e3c0762d0 0x7f4e3c078790 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4e54034000 tx=0x7f4e5402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:50.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:50.909+0000 7f4e537fe640 1 -- 172.21.15.60:0/3417173116 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4e5c151f60 con 0x7f4e5c15d3b0 2024-05-09T05:00:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.112+0000 7f4e6150a640 1 -- 172.21.15.60:0/3417173116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4e5c051040 con 0x7f4e5c15d3b0 2024-05-09T05:00:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.113+0000 7f4e537fe640 1 -- 172.21.15.60:0/3417173116 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f4e4c061670 con 0x7f4e5c15d3b0 2024-05-09T05:00:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:listed 0 entries 2024-05-09T05:00:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.114+0000 7f4e6150a640 1 -- 172.21.15.60:0/3417173116 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4e3c0762d0 msgr2=0x7f4e3c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.114+0000 7f4e6150a640 1 --2- 172.21.15.60:0/3417173116 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4e3c0762d0 0x7f4e3c078790 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4e54034000 tx=0x7f4e5402e000 comp rx=0 tx=0).stop 2024-05-09T05:00:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.114+0000 7f4e6150a640 1 -- 172.21.15.60:0/3417173116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e5c15d3b0 msgr2=0x7f4e5c1f4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.114+0000 7f4e6150a640 1 --2- 172.21.15.60:0/3417173116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e5c15d3b0 0x7f4e5c1f4d00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c007d90 tx=0x7f4e4c00e4b0 comp rx=0 tx=0).stop 2024-05-09T05:00:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.115+0000 7f4e6150a640 1 -- 172.21.15.60:0/3417173116 shutdown_connections 2024-05-09T05:00:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.115+0000 7f4e6150a640 1 --2- 172.21.15.60:0/3417173116 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4e3c0762d0 0x7f4e3c078790 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.115+0000 7f4e6150a640 1 --2- 172.21.15.60:0/3417173116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e5c15d3b0 0x7f4e5c1f4d00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.115+0000 7f4e6150a640 1 --2- 172.21.15.60:0/3417173116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e5c097de0 0x7f4e5c0ac8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.115+0000 7f4e6150a640 1 --2- 172.21.15.60:0/3417173116 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4e5c074750 0x7f4e5c15a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.115+0000 7f4e6150a640 1 -- 172.21.15.60:0/3417173116 >> 172.21.15.60:0/3417173116 conn(0x7f4e5c0972f0 msgr2=0x7f4e5c0965c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.115+0000 7f4e6150a640 1 -- 172.21.15.60:0/3417173116 shutdown_connections 2024-05-09T05:00:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.115+0000 7f4e6150a640 1 -- 172.21.15.60:0/3417173116 wait complete. 2024-05-09T05:00:51.126 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-05-09T05:00:51.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:51.127 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-05-09T05:00:51.153 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2024-05-09T05:00:51.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:51.154 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-05-09T05:00:51.210 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T05:00:51.210 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:00:51.228 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T05:00:51.228 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:00:51.285 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T05:00:51.285 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:00:51.342 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T05:00:51.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.447+0000 7fc1e25e4640 1 Processor -- start 2024-05-09T05:00:51.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.447+0000 7fc1e25e4640 1 -- start start 2024-05-09T05:00:51.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.448+0000 7fc1e25e4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc1dc06fe20 0x7fc1dc0702d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:51.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.448+0000 7fc1e25e4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1dc0736d0 0x7fc1dc073b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:51.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.448+0000 7fc1e25e4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1dc0d08e0 con 0x7fc1dc15f300 2024-05-09T05:00:51.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.448+0000 7fc1e25e4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fc1dc073240 con 0x7fc1dc06fe20 2024-05-09T05:00:51.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.448+0000 7fc1e25e4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc1dc0d0bc0 con 0x7fc1dc0736d0 2024-05-09T05:00:51.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.448+0000 7fc1db7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc1dc06fe20 0x7fc1dc0702d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:51.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.448+0000 7fc1dbfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc1dc15f300 0x7fc1dc072220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56426/0 (socket says 172.21.15.60:56426) 2024-05-09T05:00:51.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.448+0000 7fc1dbfff640 1 -- 172.21.15.60:0/646746134 learned_addr learned my addr 172.21.15.60:0/646746134 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:51.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.448+0000 7fc1e0b81640 1 --2- 172.21.15.60:0/646746134 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1dc0736d0 0x7fc1dc073b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:51.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.449+0000 7fc1db7fe640 1 -- 172.21.15.60:0/646746134 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1dc0736d0 msgr2=0x7fc1dc073b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:51.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.449+0000 7fc1db7fe640 1 --2- 172.21.15.60:0/646746134 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1dc0736d0 0x7fc1dc073b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:51.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.449+0000 7fc1db7fe640 1 -- 172.21.15.60:0/646746134 >> v1:172.21.15.60:6789/0 conn(0x7fc1dc15f300 legacy=0x7fc1dc072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:51.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.449+0000 7fc1daffd640 1 -- 172.21.15.60:0/646746134 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 316795496 0 0) 0x7fc1dc0d08e0 con 0x7fc1dc15f300 2024-05-09T05:00:51.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.449+0000 7fc1db7fe640 1 -- 172.21.15.60:0/646746134 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1dc072ed0 con 0x7fc1dc06fe20 2024-05-09T05:00:51.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.449+0000 7fc1db7fe640 1 --2- 172.21.15.60:0/646746134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc06fe20 0x7fc1dc0702d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc1d00097a0 tx=0x7fc1d002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e097002f30c1d318 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:51.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.449+0000 7fc1daffd640 1 -- 172.21.15.60:0/646746134 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d0037070 con 0x7fc1dc06fe20 2024-05-09T05:00:51.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.449+0000 7fc1daffd640 1 -- 172.21.15.60:0/646746134 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1d0032440 con 0x7fc1dc06fe20 2024-05-09T05:00:51.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.449+0000 7fc1daffd640 1 -- 172.21.15.60:0/646746134 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d0032810 con 0x7fc1dc06fe20 2024-05-09T05:00:51.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.450+0000 7fc1e25e4640 1 -- 172.21.15.60:0/646746134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc06fe20 msgr2=0x7fc1dc0702d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:51.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.450+0000 7fc1e25e4640 1 --2- 172.21.15.60:0/646746134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc06fe20 0x7fc1dc0702d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc1d00097a0 tx=0x7fc1d002a180 comp rx=0 tx=0).stop 2024-05-09T05:00:51.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.450+0000 7fc1e25e4640 1 -- 172.21.15.60:0/646746134 shutdown_connections 2024-05-09T05:00:51.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.450+0000 7fc1e25e4640 1 --2- 172.21.15.60:0/646746134 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1dc0736d0 0x7fc1dc073b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:51.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.450+0000 7fc1e25e4640 1 --2- 172.21.15.60:0/646746134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc06fe20 0x7fc1dc0702d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:51.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.450+0000 7fc1e25e4640 1 -- 172.21.15.60:0/646746134 >> 172.21.15.60:0/646746134 conn(0x7fc1dc0b4ef0 msgr2=0x7fc1dc06e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:51.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.450+0000 7fc1e25e4640 1 -- 172.21.15.60:0/646746134 shutdown_connections 2024-05-09T05:00:51.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.450+0000 7fc1e25e4640 1 -- 172.21.15.60:0/646746134 wait complete. 2024-05-09T05:00:51.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.451+0000 7fc1e25e4640 1 Processor -- start 2024-05-09T05:00:51.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.451+0000 7fc1e25e4640 1 -- start start 2024-05-09T05:00:51.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.452+0000 7fc1e25e4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1dc06fe20 0x7fc1dc0d2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:51.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.452+0000 7fc1e25e4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0736d0 0x7fc1dc0b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:51.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.452+0000 7fc1e25e4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1dc15f300 0x7fc1dc0b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:51.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.452+0000 7fc1e25e4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc0d04f0 con 0x7fc1dc0736d0 2024-05-09T05:00:51.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.452+0000 7fc1e25e4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc09c1e0 con 0x7fc1dc15f300 2024-05-09T05:00:51.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.452+0000 7fc1e25e4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc1dc0d0670 con 0x7fc1dc06fe20 2024-05-09T05:00:51.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.452+0000 7fc1db7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0736d0 0x7fc1dc0b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:51.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.452+0000 7fc1db7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0736d0 0x7fc1dc0b1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50990/0 (socket says 172.21.15.60:50990) 2024-05-09T05:00:51.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.452+0000 7fc1db7fe640 1 -- 172.21.15.60:0/2260540371 learned_addr learned my addr 172.21.15.60:0/2260540371 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:51.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.452+0000 7fc1dbfff640 1 --2- 172.21.15.60:0/2260540371 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1dc06fe20 0x7fc1dc0d2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:51.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.452+0000 7fc1e0b81640 1 --2- 172.21.15.60:0/2260540371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1dc15f300 0x7fc1dc0b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:51.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.453+0000 7fc1db7fe640 1 -- 172.21.15.60:0/2260540371 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1dc06fe20 msgr2=0x7fc1dc0d2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:51.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.453+0000 7fc1db7fe640 1 --2- 172.21.15.60:0/2260540371 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1dc06fe20 0x7fc1dc0d2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:51.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.453+0000 7fc1db7fe640 1 -- 172.21.15.60:0/2260540371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1dc15f300 msgr2=0x7fc1dc0b2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:51.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.453+0000 7fc1db7fe640 1 --2- 172.21.15.60:0/2260540371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1dc15f300 0x7fc1dc0b2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:51.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.453+0000 7fc1db7fe640 1 -- 172.21.15.60:0/2260540371 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1dc0d30b0 con 0x7fc1dc0736d0 2024-05-09T05:00:51.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.453+0000 7fc1dbfff640 1 --2- 172.21.15.60:0/2260540371 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1dc06fe20 0x7fc1dc0d2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:51.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.453+0000 7fc1e0b81640 1 --2- 172.21.15.60:0/2260540371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1dc15f300 0x7fc1dc0b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:51.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.453+0000 7fc1db7fe640 1 --2- 172.21.15.60:0/2260540371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0736d0 0x7fc1dc0b1c40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0035040 tx=0x7fc1d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:51.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.453+0000 7fc1d8ff9640 1 -- 172.21.15.60:0/2260540371 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d0037040 con 0x7fc1dc0736d0 2024-05-09T05:00:51.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.453+0000 7fc1d8ff9640 1 -- 172.21.15.60:0/2260540371 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1d0030770 con 0x7fc1dc0736d0 2024-05-09T05:00:51.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.453+0000 7fc1e25e4640 1 -- 172.21.15.60:0/2260540371 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1dc0d3390 con 0x7fc1dc0736d0 2024-05-09T05:00:51.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.453+0000 7fc1d8ff9640 1 -- 172.21.15.60:0/2260540371 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d0030b80 con 0x7fc1dc0736d0 2024-05-09T05:00:51.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.453+0000 7fc1e25e4640 1 -- 172.21.15.60:0/2260540371 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1dc0d3950 con 0x7fc1dc0736d0 2024-05-09T05:00:51.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.454+0000 7fc1e25e4640 1 -- 172.21.15.60:0/2260540371 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1dc051040 con 0x7fc1dc0736d0 2024-05-09T05:00:51.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.455+0000 7fc1d8ff9640 1 -- 172.21.15.60:0/2260540371 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc1d0044020 con 0x7fc1dc0736d0 2024-05-09T05:00:51.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.455+0000 7fc1d8ff9640 1 --2- 172.21.15.60:0/2260540371 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc1c80762d0 0x7fc1c8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:51.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.455+0000 7fc1d8ff9640 1 -- 172.21.15.60:0/2260540371 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc1d00b7db0 con 0x7fc1dc0736d0 2024-05-09T05:00:51.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.455+0000 7fc1dbfff640 1 --2- 172.21.15.60:0/2260540371 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc1c80762d0 0x7fc1c8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:51.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.456+0000 7fc1dbfff640 1 --2- 172.21.15.60:0/2260540371 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc1c80762d0 0x7fc1c8078790 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc000f60 tx=0x7fc1cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:51.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.460+0000 7fc1d8ff9640 1 -- 172.21.15.60:0/2260540371 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc1d0032e20 con 0x7fc1dc0736d0 2024-05-09T05:00:51.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.664+0000 7fc1e25e4640 1 -- 172.21.15.60:0/2260540371 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc1dc160850 con 0x7fc1dc0736d0 2024-05-09T05:00:51.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.664+0000 7fc1d8ff9640 1 -- 172.21.15.60:0/2260540371 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7fc1d0081460 con 0x7fc1dc0736d0 2024-05-09T05:00:51.666 INFO:teuthology.orchestra.run.smithi060.stderr:listed 0 entries 2024-05-09T05:00:51.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.666+0000 7fc1e25e4640 1 -- 172.21.15.60:0/2260540371 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc1c80762d0 msgr2=0x7fc1c8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:51.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.666+0000 7fc1e25e4640 1 --2- 172.21.15.60:0/2260540371 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc1c80762d0 0x7fc1c8078790 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc000f60 tx=0x7fc1cc009040 comp rx=0 tx=0).stop 2024-05-09T05:00:51.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.666+0000 7fc1e25e4640 1 -- 172.21.15.60:0/2260540371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0736d0 msgr2=0x7fc1dc0b1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:51.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.666+0000 7fc1e25e4640 1 --2- 172.21.15.60:0/2260540371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0736d0 0x7fc1dc0b1c40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0035040 tx=0x7fc1d00046c0 comp rx=0 tx=0).stop 2024-05-09T05:00:51.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.667+0000 7fc1e25e4640 1 -- 172.21.15.60:0/2260540371 shutdown_connections 2024-05-09T05:00:51.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.667+0000 7fc1e25e4640 1 --2- 172.21.15.60:0/2260540371 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc1c80762d0 0x7fc1c8078790 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:51.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.667+0000 7fc1e25e4640 1 --2- 172.21.15.60:0/2260540371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1dc15f300 0x7fc1dc0b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:51.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.667+0000 7fc1e25e4640 1 --2- 172.21.15.60:0/2260540371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1dc0736d0 0x7fc1dc0b1c40 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:51.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.667+0000 7fc1e25e4640 1 --2- 172.21.15.60:0/2260540371 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc1dc06fe20 0x7fc1dc0d2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:51.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.667+0000 7fc1e25e4640 1 -- 172.21.15.60:0/2260540371 >> 172.21.15.60:0/2260540371 conn(0x7fc1dc0b4ef0 msgr2=0x7fc1dc162310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:51.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.667+0000 7fc1e25e4640 1 -- 172.21.15.60:0/2260540371 shutdown_connections 2024-05-09T05:00:51.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.667+0000 7fc1e25e4640 1 -- 172.21.15.60:0/2260540371 wait complete. 2024-05-09T05:00:51.678 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-05-09T05:00:51.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:51.679 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-05-09T05:00:51.710 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T05:00:51.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:00:51.710 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-05-09T05:00:51.767 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-09T05:00:51.767 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:00:51.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.871+0000 7f819de45640 1 Processor -- start 2024-05-09T05:00:51.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.872+0000 7f819de45640 1 -- start start 2024-05-09T05:00:51.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.872+0000 7f819de45640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f819815f3c0 0x7f819809ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:51.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.872+0000 7f819de45640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f819815f7a0 con 0x7f81980b4d30 2024-05-09T05:00:51.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.872+0000 7f819de45640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f819809f480 con 0x7f8198073fb0 2024-05-09T05:00:51.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.872+0000 7f819de45640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f8198071cb0 con 0x7f819815f3c0 2024-05-09T05:00:51.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.872+0000 7f8197fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f81980b4d30 0x7f8198072140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56428/0 (socket says 172.21.15.60:56428) 2024-05-09T05:00:51.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.872+0000 7f8197fff640 1 -- 172.21.15.60:0/3868286500 learned_addr learned my addr 172.21.15.60:0/3868286500 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:51.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.873+0000 7f81977fe640 1 --2- 172.21.15.60:0/3868286500 >> v2:172.21.15.114:3300/0 conn(0x7f819815f3c0 0x7f819809ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:51.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.873+0000 7f8196ffd640 1 -- 172.21.15.60:0/3868286500 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3070641742 0 0) 0x7f819815f7a0 con 0x7f81980b4d30 2024-05-09T05:00:51.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.873+0000 7f8196ffd640 1 -- 172.21.15.60:0/3868286500 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8184003520 con 0x7f81980b4d30 2024-05-09T05:00:51.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.873+0000 7f81977fe640 1 -- 172.21.15.60:0/3868286500 >> v1:172.21.15.114:6789/0 conn(0x7f8198073fb0 legacy=0x7f81980b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:51.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.873+0000 7f81977fe640 1 -- 172.21.15.60:0/3868286500 >> v1:172.21.15.60:6789/0 conn(0x7f81980b4d30 legacy=0x7f8198072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:51.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.873+0000 7f81977fe640 1 -- 172.21.15.60:0/3868286500 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f819809f760 con 0x7f819815f3c0 2024-05-09T05:00:51.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.874+0000 7f81977fe640 1 --2- 172.21.15.60:0/3868286500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f819815f3c0 0x7f819809ebe0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f81880097a0 tx=0x7f818802a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e1806e29e78978d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:51.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.874+0000 7f8196ffd640 1 -- 172.21.15.60:0/3868286500 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8188037070 con 0x7f819815f3c0 2024-05-09T05:00:51.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.874+0000 7f8196ffd640 1 -- 172.21.15.60:0/3868286500 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8188032440 con 0x7f819815f3c0 2024-05-09T05:00:51.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.874+0000 7f8196ffd640 1 -- 172.21.15.60:0/3868286500 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81880327f0 con 0x7f819815f3c0 2024-05-09T05:00:51.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.874+0000 7f819de45640 1 -- 172.21.15.60:0/3868286500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f819815f3c0 msgr2=0x7f819809ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:51.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.874+0000 7f819de45640 1 --2- 172.21.15.60:0/3868286500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f819815f3c0 0x7f819809ebe0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f81880097a0 tx=0x7f818802a1a0 comp rx=0 tx=0).stop 2024-05-09T05:00:51.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.875+0000 7f819de45640 1 -- 172.21.15.60:0/3868286500 shutdown_connections 2024-05-09T05:00:51.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.875+0000 7f819de45640 1 --2- 172.21.15.60:0/3868286500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f819815f3c0 0x7f819809ebe0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:51.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.875+0000 7f819de45640 1 -- 172.21.15.60:0/3868286500 >> 172.21.15.60:0/3868286500 conn(0x7f81980d1470 msgr2=0x7f81980d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:51.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.875+0000 7f819de45640 1 -- 172.21.15.60:0/3868286500 shutdown_connections 2024-05-09T05:00:51.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.875+0000 7f819de45640 1 -- 172.21.15.60:0/3868286500 wait complete. 2024-05-09T05:00:51.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.876+0000 7f819de45640 1 Processor -- start 2024-05-09T05:00:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.876+0000 7f819de45640 1 -- start start 2024-05-09T05:00:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.876+0000 7f819de45640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8198073fb0 0x7f81981f4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.876+0000 7f819de45640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81980b4d30 0x7f81981f4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.876+0000 7f819de45640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f819815f3c0 0x7f81980affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.876+0000 7f819de45640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8198072850 con 0x7f8198073fb0 2024-05-09T05:00:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.876+0000 7f819de45640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f819809f120 con 0x7f81980b4d30 2024-05-09T05:00:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.876+0000 7f819de45640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f81980729d0 con 0x7f819815f3c0 2024-05-09T05:00:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.877+0000 7f81977fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8198073fb0 0x7f81981f4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.877+0000 7f81977fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8198073fb0 0x7f81981f4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51006/0 (socket says 172.21.15.60:51006) 2024-05-09T05:00:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.877+0000 7f81977fe640 1 -- 172.21.15.60:0/2431554023 learned_addr learned my addr 172.21.15.60:0/2431554023 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.877+0000 7f818ffff640 1 --2- 172.21.15.60:0/2431554023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81980b4d30 0x7f81981f4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.877+0000 7f81977fe640 1 -- 172.21.15.60:0/2431554023 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f819815f3c0 msgr2=0x7f81980affa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.877+0000 7f81977fe640 1 --2- 172.21.15.60:0/2431554023 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f819815f3c0 0x7f81980affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.877+0000 7f81977fe640 1 -- 172.21.15.60:0/2431554023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81980b4d30 msgr2=0x7f81981f4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.877+0000 7f81977fe640 1 --2- 172.21.15.60:0/2431554023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81980b4d30 0x7f81981f4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.877+0000 7f81977fe640 1 -- 172.21.15.60:0/2431554023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81981f5150 con 0x7f8198073fb0 2024-05-09T05:00:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.877+0000 7f81977fe640 1 --2- 172.21.15.60:0/2431554023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8198073fb0 0x7f81981f4550 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8188035040 tx=0x7f81880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.877+0000 7f8194ff9640 1 -- 172.21.15.60:0/2431554023 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8188037050 con 0x7f8198073fb0 2024-05-09T05:00:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.877+0000 7f8194ff9640 1 -- 172.21.15.60:0/2431554023 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8188038040 con 0x7f8198073fb0 2024-05-09T05:00:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.877+0000 7f819de45640 1 -- 172.21.15.60:0/2431554023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81981f5430 con 0x7f8198073fb0 2024-05-09T05:00:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.877+0000 7f8194ff9640 1 -- 172.21.15.60:0/2431554023 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8188004450 con 0x7f8198073fb0 2024-05-09T05:00:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.877+0000 7f819de45640 1 -- 172.21.15.60:0/2431554023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81980b04e0 con 0x7f8198073fb0 2024-05-09T05:00:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.878+0000 7f819de45640 1 -- 172.21.15.60:0/2431554023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8198051040 con 0x7f8198073fb0 2024-05-09T05:00:51.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.879+0000 7f8194ff9640 1 -- 172.21.15.60:0/2431554023 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f818803c6d0 con 0x7f8198073fb0 2024-05-09T05:00:51.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.880+0000 7f8194ff9640 1 --2- 172.21.15.60:0/2431554023 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f81780762d0 0x7f8178078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:51.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.880+0000 7f8194ff9640 1 -- 172.21.15.60:0/2431554023 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f81880b8400 con 0x7f8198073fb0 2024-05-09T05:00:51.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.880+0000 7f818ffff640 1 --2- 172.21.15.60:0/2431554023 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f81780762d0 0x7f8178078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:51.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.880+0000 7f818ffff640 1 --2- 172.21.15.60:0/2431554023 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f81780762d0 0x7f8178078790 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f81980acdc0 tx=0x7f8180009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:51.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:51.884+0000 7f8194ff9640 1 -- 172.21.15.60:0/2431554023 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8188081a20 con 0x7f8198073fb0 2024-05-09T05:00:52.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.115+0000 7f819de45640 1 -- 172.21.15.60:0/2431554023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81980687c0 con 0x7f8198073fb0 2024-05-09T05:00:52.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.116+0000 7f8194ff9640 1 -- 172.21.15.60:0/2431554023 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7f8188081320 con 0x7f8198073fb0 2024-05-09T05:00:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:52.117 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1285340937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1285340937},{"type":"v1","addr":"172.21.15.60:6833","nonce":1285340937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2599193390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2599193390},{"type":"v1","addr":"172.21.15.114:6835","nonce":2599193390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3432609307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3432609307},{"type":"v1","addr":"172.21.15.114:6837","nonce":3432609307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:00:12.208556+0000","modified":"2024-05-09T05:00:31.347095+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2135423314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2135423314},{"type":"v1","addr":"172.21.15.60:6835","nonce":2135423314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-05-09T05:00:52.117 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2024-05-09T05:00:52.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.118+0000 7f819de45640 1 -- 172.21.15.60:0/2431554023 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f81780762d0 msgr2=0x7f8178078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:52.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.118+0000 7f819de45640 1 --2- 172.21.15.60:0/2431554023 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f81780762d0 0x7f8178078790 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f81980acdc0 tx=0x7f8180009040 comp rx=0 tx=0).stop 2024-05-09T05:00:52.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.118+0000 7f819de45640 1 -- 172.21.15.60:0/2431554023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8198073fb0 msgr2=0x7f81981f4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:52.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.118+0000 7f819de45640 1 --2- 172.21.15.60:0/2431554023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8198073fb0 0x7f81981f4550 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8188035040 tx=0x7f81880046c0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.118+0000 7f819de45640 1 -- 172.21.15.60:0/2431554023 shutdown_connections 2024-05-09T05:00:52.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.118+0000 7f819de45640 1 --2- 172.21.15.60:0/2431554023 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f81780762d0 0x7f8178078790 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.118+0000 7f819de45640 1 --2- 172.21.15.60:0/2431554023 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f819815f3c0 0x7f81980affa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.118+0000 7f819de45640 1 --2- 172.21.15.60:0/2431554023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81980b4d30 0x7f81981f4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.118+0000 7f819de45640 1 --2- 172.21.15.60:0/2431554023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8198073fb0 0x7f81981f4550 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.118+0000 7f819de45640 1 -- 172.21.15.60:0/2431554023 >> 172.21.15.60:0/2431554023 conn(0x7f81980d1470 msgr2=0x7f819809e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:52.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.118+0000 7f819de45640 1 -- 172.21.15.60:0/2431554023 shutdown_connections 2024-05-09T05:00:52.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.118+0000 7f819de45640 1 -- 172.21.15.60:0/2431554023 wait complete. 2024-05-09T05:00:52.130 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:00:52.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.234+0000 7fa3b70e1640 1 Processor -- start 2024-05-09T05:00:52.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.234+0000 7fa3b70e1640 1 -- start start 2024-05-09T05:00:52.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.234+0000 7fa3b70e1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa3b00d1050 0x7fa3b00ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:52.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.234+0000 7fa3b70e1640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa3b0096eb0 0x7fa3b009b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:52.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.234+0000 7fa3b70e1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3b009be90 0x7fa3b00d1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:52.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.234+0000 7fa3b70e1640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fa3b00af370 con 0x7fa3b00d1050 2024-05-09T05:00:52.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.234+0000 7fa3b70e1640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fa3b0096a00 con 0x7fa3b0096eb0 2024-05-09T05:00:52.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.234+0000 7fa3b70e1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa3b00d2dd0 con 0x7fa3b009be90 2024-05-09T05:00:52.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.234+0000 7fa3b4e7d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa3b00d1050 0x7fa3b00ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.235+0000 7fa3b4e7d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa3b00d1050 0x7fa3b00ae9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51012/0 (socket says 172.21.15.60:51012) 2024-05-09T05:00:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.235+0000 7fa3b4e7d640 1 -- 172.21.15.60:0/2305456903 learned_addr learned my addr 172.21.15.60:0/2305456903 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.235+0000 7fa3affff640 1 --2- 172.21.15.60:0/2305456903 >> v2:172.21.15.114:3300/0 conn(0x7fa3b0096eb0 0x7fa3b009b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.235+0000 7fa3b567e640 1 --2- 172.21.15.60:0/2305456903 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3b009be90 0x7fa3b00d1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.235+0000 7fa3b4e7d640 1 -- 172.21.15.60:0/2305456903 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3b009be90 msgr2=0x7fa3b00d1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.235+0000 7fa3b4e7d640 1 --2- 172.21.15.60:0/2305456903 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3b009be90 0x7fa3b00d1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.235+0000 7fa3b4e7d640 1 -- 172.21.15.60:0/2305456903 >> v2:172.21.15.114:3300/0 conn(0x7fa3b0096eb0 msgr2=0x7fa3b009b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.235+0000 7fa3b4e7d640 1 --2- 172.21.15.60:0/2305456903 >> v2:172.21.15.114:3300/0 conn(0x7fa3b0096eb0 0x7fa3b009b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.235+0000 7fa3b4e7d640 1 -- 172.21.15.60:0/2305456903 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b00d30b0 con 0x7fa3b00d1050 2024-05-09T05:00:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.235+0000 7fa3b4e7d640 1 --2- 172.21.15.60:0/2305456903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3b00d1050 0x7fa3b00ae9f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa3b00993a0 tx=0x7fa3a002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8c4232f26b6f97b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:52.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.236+0000 7fa3af7fe640 1 -- 172.21.15.60:0/2305456903 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0037070 con 0x7fa3b00d1050 2024-05-09T05:00:52.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.236+0000 7fa3af7fe640 1 -- 172.21.15.60:0/2305456903 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3a0032440 con 0x7fa3b00d1050 2024-05-09T05:00:52.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.236+0000 7fa3af7fe640 1 -- 172.21.15.60:0/2305456903 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0032810 con 0x7fa3b00d1050 2024-05-09T05:00:52.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.236+0000 7fa3b70e1640 1 -- 172.21.15.60:0/2305456903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3b00d1050 msgr2=0x7fa3b00ae9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:52.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.236+0000 7fa3b70e1640 1 --2- 172.21.15.60:0/2305456903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3b00d1050 0x7fa3b00ae9f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa3b00993a0 tx=0x7fa3a002a1e0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.236+0000 7fa3b70e1640 1 -- 172.21.15.60:0/2305456903 shutdown_connections 2024-05-09T05:00:52.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.236+0000 7fa3b70e1640 1 --2- 172.21.15.60:0/2305456903 >> v2:172.21.15.114:3300/0 conn(0x7fa3b0096eb0 0x7fa3b009b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.236+0000 7fa3b70e1640 1 --2- 172.21.15.60:0/2305456903 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3b009be90 0x7fa3b00d1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.236+0000 7fa3b70e1640 1 --2- 172.21.15.60:0/2305456903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3b00d1050 0x7fa3b00ae9f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.236+0000 7fa3b70e1640 1 -- 172.21.15.60:0/2305456903 >> 172.21.15.60:0/2305456903 conn(0x7fa3b00993f0 msgr2=0x7fa3b0099800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:52.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.236+0000 7fa3b70e1640 1 -- 172.21.15.60:0/2305456903 shutdown_connections 2024-05-09T05:00:52.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.236+0000 7fa3b70e1640 1 -- 172.21.15.60:0/2305456903 wait complete. 2024-05-09T05:00:52.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.237+0000 7fa3b70e1640 1 Processor -- start 2024-05-09T05:00:52.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.238+0000 7fa3b70e1640 1 -- start start 2024-05-09T05:00:52.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.238+0000 7fa3b70e1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3b0096eb0 0x7fa3b006f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:52.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.238+0000 7fa3b70e1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3b009be90 0x7fa3b0092f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:52.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.238+0000 7fa3b70e1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3b00d1050 0x7fa3b00934b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:52.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.238+0000 7fa3b70e1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b00af0b0 con 0x7fa3b0096eb0 2024-05-09T05:00:52.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.238+0000 7fa3b70e1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b00aef30 con 0x7fa3b009be90 2024-05-09T05:00:52.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.238+0000 7fa3b70e1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa3b00d1970 con 0x7fa3b00d1050 2024-05-09T05:00:52.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.239+0000 7fa3b4e7d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3b0096eb0 0x7fa3b006f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:52.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.239+0000 7fa3b4e7d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3b0096eb0 0x7fa3b006f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51026/0 (socket says 172.21.15.60:51026) 2024-05-09T05:00:52.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.239+0000 7fa3b4e7d640 1 -- 172.21.15.60:0/997391347 learned_addr learned my addr 172.21.15.60:0/997391347 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:52.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.239+0000 7fa3affff640 1 --2- 172.21.15.60:0/997391347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3b009be90 0x7fa3b0092f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:52.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.239+0000 7fa3b567e640 1 --2- 172.21.15.60:0/997391347 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3b00d1050 0x7fa3b00934b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:52.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.239+0000 7fa3b4e7d640 1 -- 172.21.15.60:0/997391347 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3b00d1050 msgr2=0x7fa3b00934b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:52.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.239+0000 7fa3b4e7d640 1 --2- 172.21.15.60:0/997391347 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3b00d1050 0x7fa3b00934b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.239+0000 7fa3b4e7d640 1 -- 172.21.15.60:0/997391347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3b009be90 msgr2=0x7fa3b0092f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:52.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.239+0000 7fa3b4e7d640 1 --2- 172.21.15.60:0/997391347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3b009be90 0x7fa3b0092f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.239+0000 7fa3b4e7d640 1 -- 172.21.15.60:0/997391347 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b0095460 con 0x7fa3b0096eb0 2024-05-09T05:00:52.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.239+0000 7fa3b567e640 1 --2- 172.21.15.60:0/997391347 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3b00d1050 0x7fa3b00934b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:52.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.239+0000 7fa3b4e7d640 1 --2- 172.21.15.60:0/997391347 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3b0096eb0 0x7fa3b006f9d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa3a002a220 tx=0x7fa3a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:52.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.239+0000 7fa3ad7fa640 1 -- 172.21.15.60:0/997391347 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0041020 con 0x7fa3b0096eb0 2024-05-09T05:00:52.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.239+0000 7fa3ad7fa640 1 -- 172.21.15.60:0/997391347 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3a00306c0 con 0x7fa3b0096eb0 2024-05-09T05:00:52.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.239+0000 7fa3b70e1640 1 -- 172.21.15.60:0/997391347 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3b0095740 con 0x7fa3b0096eb0 2024-05-09T05:00:52.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.239+0000 7fa3ad7fa640 1 -- 172.21.15.60:0/997391347 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0037040 con 0x7fa3b0096eb0 2024-05-09T05:00:52.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.239+0000 7fa3b70e1640 1 -- 172.21.15.60:0/997391347 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3b0095d00 con 0x7fa3b0096eb0 2024-05-09T05:00:52.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.240+0000 7fa3b70e1640 1 -- 172.21.15.60:0/997391347 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3b0075540 con 0x7fa3b0096eb0 2024-05-09T05:00:52.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.241+0000 7fa3ad7fa640 1 -- 172.21.15.60:0/997391347 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa3a0044020 con 0x7fa3b0096eb0 2024-05-09T05:00:52.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.241+0000 7fa3ad7fa640 1 --2- 172.21.15.60:0/997391347 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa3900762d0 0x7fa390078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:52.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.242+0000 7fa3ad7fa640 1 -- 172.21.15.60:0/997391347 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa3a00b7ff0 con 0x7fa3b0096eb0 2024-05-09T05:00:52.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.242+0000 7fa3affff640 1 --2- 172.21.15.60:0/997391347 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa3900762d0 0x7fa390078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:52.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.242+0000 7fa3affff640 1 --2- 172.21.15.60:0/997391347 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa3900762d0 0x7fa390078790 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0093c30 tx=0x7fa398008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:52.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.246+0000 7fa3ad7fa640 1 -- 172.21.15.60:0/997391347 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa3a004b030 con 0x7fa3b0096eb0 2024-05-09T05:00:52.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.478+0000 7fa3b70e1640 1 -- 172.21.15.60:0/997391347 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3b009b450 con 0x7fa3b0096eb0 2024-05-09T05:00:52.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.478+0000 7fa3ad7fa640 1 -- 172.21.15.60:0/997391347 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7fa3a0081660 con 0x7fa3b0096eb0 2024-05-09T05:00:52.483 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:52.483 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1285340937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1285340937},{"type":"v1","addr":"172.21.15.60:6833","nonce":1285340937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2599193390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2599193390},{"type":"v1","addr":"172.21.15.114:6835","nonce":2599193390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3432609307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3432609307},{"type":"v1","addr":"172.21.15.114:6837","nonce":3432609307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:00:12.208556+0000","modified":"2024-05-09T05:00:31.347095+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2135423314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2135423314},{"type":"v1","addr":"172.21.15.60:6835","nonce":2135423314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-05-09T05:00:52.484 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2024-05-09T05:00:52.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.480+0000 7fa3b70e1640 1 -- 172.21.15.60:0/997391347 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa3900762d0 msgr2=0x7fa390078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:52.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.480+0000 7fa3b70e1640 1 --2- 172.21.15.60:0/997391347 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa3900762d0 0x7fa390078790 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0093c30 tx=0x7fa398008040 comp rx=0 tx=0).stop 2024-05-09T05:00:52.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.480+0000 7fa3b70e1640 1 -- 172.21.15.60:0/997391347 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3b0096eb0 msgr2=0x7fa3b006f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:52.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.480+0000 7fa3b70e1640 1 --2- 172.21.15.60:0/997391347 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3b0096eb0 0x7fa3b006f9d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa3a002a220 tx=0x7fa3a00046c0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.481+0000 7fa3b70e1640 1 -- 172.21.15.60:0/997391347 shutdown_connections 2024-05-09T05:00:52.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.481+0000 7fa3b70e1640 1 --2- 172.21.15.60:0/997391347 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa3900762d0 0x7fa390078790 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.481+0000 7fa3b70e1640 1 --2- 172.21.15.60:0/997391347 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa3b00d1050 0x7fa3b00934b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.481+0000 7fa3b70e1640 1 --2- 172.21.15.60:0/997391347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3b009be90 0x7fa3b0092f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.481+0000 7fa3b70e1640 1 --2- 172.21.15.60:0/997391347 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3b0096eb0 0x7fa3b006f9d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.481+0000 7fa3b70e1640 1 -- 172.21.15.60:0/997391347 >> 172.21.15.60:0/997391347 conn(0x7fa3b00993f0 msgr2=0x7fa3b00af4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:52.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.481+0000 7fa3b70e1640 1 -- 172.21.15.60:0/997391347 shutdown_connections 2024-05-09T05:00:52.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.481+0000 7fa3b70e1640 1 -- 172.21.15.60:0/997391347 wait complete. 2024-05-09T05:00:52.493 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:00:52.512 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:00:52.539 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:00:52.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.599+0000 7f88d8189640 1 Processor -- start 2024-05-09T05:00:52.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.599+0000 7f88d8189640 1 -- start start 2024-05-09T05:00:52.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.600+0000 7f88d8189640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f88d00735a0 0x7f88d00ad010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:52.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.600+0000 7f88d8189640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f88d00ad550 0x7f88d006f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:52.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.600+0000 7f88d8189640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88d015fcf0 con 0x7f88d006f590 2024-05-09T05:00:52.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.600+0000 7f88d8189640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f88d006fa00 con 0x7f88d00ad550 2024-05-09T05:00:52.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.600+0000 7f88d8189640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f88d00aec60 con 0x7f88d00735a0 2024-05-09T05:00:52.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.600+0000 7f88d5724640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f88d00ad550 0x7f88d006f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:52.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.600+0000 7f88d6726640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f88d006f590 0x7f88d0162ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56444/0 (socket says 172.21.15.60:56444) 2024-05-09T05:00:52.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.600+0000 7f88d6726640 1 -- 172.21.15.60:0/1935208763 learned_addr learned my addr 172.21.15.60:0/1935208763 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:52.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.600+0000 7f88d5724640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f88d00ad550 0x7f88d006f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51028/0 (socket says 172.21.15.60:51028) 2024-05-09T05:00:52.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.600+0000 7f88d5724640 1 -- 172.21.15.60:0/1935208763 >> v2:172.21.15.114:3300/0 conn(0x7f88d00735a0 msgr2=0x7f88d00ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:52.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.600+0000 7f88d5724640 1 --2- 172.21.15.60:0/1935208763 >> v2:172.21.15.114:3300/0 conn(0x7f88d00735a0 0x7f88d00ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.600+0000 7f88d5724640 1 -- 172.21.15.60:0/1935208763 >> v1:172.21.15.60:6789/0 conn(0x7f88d006f590 legacy=0x7f88d0162ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:52.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.600+0000 7f88d4f23640 1 -- 172.21.15.60:0/1935208763 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3033585162 0 0) 0x7f88d015fcf0 con 0x7f88d006f590 2024-05-09T05:00:52.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.600+0000 7f88d5724640 1 -- 172.21.15.60:0/1935208763 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d015ffd0 con 0x7f88d00ad550 2024-05-09T05:00:52.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.601+0000 7f88d5724640 1 --2- 172.21.15.60:0/1935208763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d00ad550 0x7f88d006f050 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f88c00097a0 tx=0x7f88c002a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a549e7d49922091 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:52.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.602+0000 7f88d4f23640 1 -- 172.21.15.60:0/1935208763 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c0037070 con 0x7f88d00ad550 2024-05-09T05:00:52.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.602+0000 7f88d4f23640 1 -- 172.21.15.60:0/1935208763 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88c002adc0 con 0x7f88d00ad550 2024-05-09T05:00:52.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.602+0000 7f88d4f23640 1 -- 172.21.15.60:0/1935208763 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c0032660 con 0x7f88d00ad550 2024-05-09T05:00:52.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.602+0000 7f88d8189640 1 -- 172.21.15.60:0/1935208763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d00ad550 msgr2=0x7f88d006f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:52.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.602+0000 7f88d8189640 1 --2- 172.21.15.60:0/1935208763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d00ad550 0x7f88d006f050 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f88c00097a0 tx=0x7f88c002a0b0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.602+0000 7f88d8189640 1 -- 172.21.15.60:0/1935208763 shutdown_connections 2024-05-09T05:00:52.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.602+0000 7f88d8189640 1 --2- 172.21.15.60:0/1935208763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d00ad550 0x7f88d006f050 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.602+0000 7f88d8189640 1 --2- 172.21.15.60:0/1935208763 >> v2:172.21.15.114:3300/0 conn(0x7f88d00735a0 0x7f88d00ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.602+0000 7f88d8189640 1 -- 172.21.15.60:0/1935208763 >> 172.21.15.60:0/1935208763 conn(0x7f88d006d330 msgr2=0x7f88d00b29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:52.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.602+0000 7f88d8189640 1 -- 172.21.15.60:0/1935208763 shutdown_connections 2024-05-09T05:00:52.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.602+0000 7f88d8189640 1 -- 172.21.15.60:0/1935208763 wait complete. 2024-05-09T05:00:52.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.603+0000 7f88d8189640 1 Processor -- start 2024-05-09T05:00:52.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.604+0000 7f88d8189640 1 -- start start 2024-05-09T05:00:52.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.604+0000 7f88d8189640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d006f590 0x7f88d014c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:52.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.604+0000 7f88d8189640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88d00735a0 0x7f88d01f4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:52.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.604+0000 7f88d8189640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88d00ad550 0x7f88d0161d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:52.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.604+0000 7f88d8189640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f88d0160590 con 0x7f88d006f590 2024-05-09T05:00:52.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.604+0000 7f88d8189640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f88d0160870 con 0x7f88d00ad550 2024-05-09T05:00:52.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.604+0000 7f88d8189640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f88d015f8e0 con 0x7f88d00735a0 2024-05-09T05:00:52.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.604+0000 7f88d5f25640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d006f590 0x7f88d014c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:52.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.604+0000 7f88d5f25640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d006f590 0x7f88d014c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51042/0 (socket says 172.21.15.60:51042) 2024-05-09T05:00:52.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.604+0000 7f88d5f25640 1 -- 172.21.15.60:0/3735046481 learned_addr learned my addr 172.21.15.60:0/3735046481 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:52.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.605+0000 7f88d6726640 1 --2- 172.21.15.60:0/3735046481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88d00ad550 0x7f88d0161d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:52.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.605+0000 7f88d5724640 1 --2- 172.21.15.60:0/3735046481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88d00735a0 0x7f88d01f4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:52.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.605+0000 7f88d5f25640 1 -- 172.21.15.60:0/3735046481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88d00735a0 msgr2=0x7f88d01f4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:52.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.605+0000 7f88d5f25640 1 --2- 172.21.15.60:0/3735046481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88d00735a0 0x7f88d01f4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.605+0000 7f88d5f25640 1 -- 172.21.15.60:0/3735046481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88d00ad550 msgr2=0x7f88d0161d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:52.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.605+0000 7f88d5f25640 1 --2- 172.21.15.60:0/3735046481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88d00ad550 0x7f88d0161d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.605+0000 7f88d5f25640 1 -- 172.21.15.60:0/3735046481 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d015cb50 con 0x7f88d006f590 2024-05-09T05:00:52.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.605+0000 7f88d5f25640 1 --2- 172.21.15.60:0/3735046481 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d006f590 0x7f88d014c4f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f88cc00b4a0 tx=0x7f88cc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:52.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.605+0000 7f88c67fc640 1 -- 172.21.15.60:0/3735046481 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88cc014070 con 0x7f88d006f590 2024-05-09T05:00:52.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.605+0000 7f88c67fc640 1 -- 172.21.15.60:0/3735046481 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88cc010610 con 0x7f88d006f590 2024-05-09T05:00:52.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.605+0000 7f88d8189640 1 -- 172.21.15.60:0/3735046481 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88d015a1b0 con 0x7f88d006f590 2024-05-09T05:00:52.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.605+0000 7f88c67fc640 1 -- 172.21.15.60:0/3735046481 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88cc010a00 con 0x7f88d006f590 2024-05-09T05:00:52.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.605+0000 7f88d8189640 1 -- 172.21.15.60:0/3735046481 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88d01f5430 con 0x7f88d006f590 2024-05-09T05:00:52.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.606+0000 7f88d8189640 1 -- 172.21.15.60:0/3735046481 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88d0159fa0 con 0x7f88d006f590 2024-05-09T05:00:52.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.607+0000 7f88c67fc640 1 -- 172.21.15.60:0/3735046481 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f88cc017070 con 0x7f88d006f590 2024-05-09T05:00:52.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.608+0000 7f88c67fc640 1 --2- 172.21.15.60:0/3735046481 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f88b4076300 0x7f88b40787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:52.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.608+0000 7f88c67fc640 1 -- 172.21.15.60:0/3735046481 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f88cc097230 con 0x7f88d006f590 2024-05-09T05:00:52.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.608+0000 7f88d5724640 1 --2- 172.21.15.60:0/3735046481 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f88b4076300 0x7f88b40787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:52.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.608+0000 7f88d5724640 1 --2- 172.21.15.60:0/3735046481 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f88b4076300 0x7f88b40787c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f88c0034000 tx=0x7f88c002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:52.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.616+0000 7f88c67fc640 1 -- 172.21.15.60:0/3735046481 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f88cc060200 con 0x7f88d006f590 2024-05-09T05:00:52.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.809+0000 7f88d8189640 1 -- 172.21.15.60:0/3735046481 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88d0051040 con 0x7f88d006f590 2024-05-09T05:00:52.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.811+0000 7f88c67fc640 1 -- 172.21.15.60:0/3735046481 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18490 (secure 0 0 0) 0x7f88cc064500 con 0x7f88d006f590 2024-05-09T05:00:52.812 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:52.812 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:00:17.248958+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:00:10.358133+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:00:11.157769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T05:00:14.850459+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:00:52.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.813+0000 7f88d8189640 1 -- 172.21.15.60:0/3735046481 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f88b4076300 msgr2=0x7f88b40787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:52.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.813+0000 7f88d8189640 1 --2- 172.21.15.60:0/3735046481 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f88b4076300 0x7f88b40787c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f88c0034000 tx=0x7f88c002e000 comp rx=0 tx=0).stop 2024-05-09T05:00:52.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.813+0000 7f88d8189640 1 -- 172.21.15.60:0/3735046481 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d006f590 msgr2=0x7f88d014c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:52.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.813+0000 7f88d8189640 1 --2- 172.21.15.60:0/3735046481 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d006f590 0x7f88d014c4f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f88cc00b4a0 tx=0x7f88cc00b960 comp rx=0 tx=0).stop 2024-05-09T05:00:52.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.813+0000 7f88d8189640 1 -- 172.21.15.60:0/3735046481 shutdown_connections 2024-05-09T05:00:52.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.813+0000 7f88d8189640 1 --2- 172.21.15.60:0/3735046481 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f88b4076300 0x7f88b40787c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.813+0000 7f88d8189640 1 --2- 172.21.15.60:0/3735046481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88d00ad550 0x7f88d0161d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.813+0000 7f88d8189640 1 --2- 172.21.15.60:0/3735046481 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88d00735a0 0x7f88d01f4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.813+0000 7f88d8189640 1 --2- 172.21.15.60:0/3735046481 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f88d006f590 0x7f88d014c4f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.813+0000 7f88d8189640 1 -- 172.21.15.60:0/3735046481 >> 172.21.15.60:0/3735046481 conn(0x7f88d006d330 msgr2=0x7f88d00b36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:52.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.813+0000 7f88d8189640 1 -- 172.21.15.60:0/3735046481 shutdown_connections 2024-05-09T05:00:52.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.813+0000 7f88d8189640 1 -- 172.21.15.60:0/3735046481 wait complete. 2024-05-09T05:00:52.825 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T05:00:52.826 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:00:52.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.972+0000 7f4ccaf4d640 1 Processor -- start 2024-05-09T05:00:52.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.972+0000 7f4ccaf4d640 1 -- start start 2024-05-09T05:00:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.973+0000 7f4ccaf4d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f4cc406f9f0 0x7f4cc406ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.973+0000 7f4ccaf4d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cc406ff60 con 0x7f4cc41606f0 2024-05-09T05:00:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.973+0000 7f4ccaf4d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cc4070240 con 0x7f4cc415f300 2024-05-09T05:00:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.973+0000 7f4ccaf4d640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f4cc4070520 con 0x7f4cc406f9f0 2024-05-09T05:00:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.973+0000 7f4cc3fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4cc41606f0 0x7f4cc40d33f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56452/0 (socket says 172.21.15.60:56452) 2024-05-09T05:00:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.973+0000 7f4cc3fff640 1 -- 172.21.15.60:0/3481252601 learned_addr learned my addr 172.21.15.60:0/3481252601 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.974+0000 7f4cc94ea640 1 --2- 172.21.15.60:0/3481252601 >> v2:172.21.15.114:3300/0 conn(0x7f4cc406f9f0 0x7f4cc406ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.974+0000 7f4cc37fe640 1 -- 172.21.15.60:0/3481252601 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1390803589 0 0) 0x7f4cc406ff60 con 0x7f4cc41606f0 2024-05-09T05:00:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.974+0000 7f4cc37fe640 1 -- 172.21.15.60:0/3481252601 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cb4003520 con 0x7f4cc41606f0 2024-05-09T05:00:52.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.974+0000 7f4cc37fe640 1 -- 172.21.15.60:0/3481252601 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3143920214 0 0) 0x7f4cc4070240 con 0x7f4cc415f300 2024-05-09T05:00:52.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.974+0000 7f4cc37fe640 1 -- 172.21.15.60:0/3481252601 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cc406ff60 con 0x7f4cc415f300 2024-05-09T05:00:52.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.974+0000 7f4cc94ea640 1 -- 172.21.15.60:0/3481252601 >> v1:172.21.15.114:6789/0 conn(0x7f4cc415f300 legacy=0x7f4cc41639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:52.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.974+0000 7f4cc94ea640 1 -- 172.21.15.60:0/3481252601 >> v1:172.21.15.60:6789/0 conn(0x7f4cc41606f0 legacy=0x7f4cc40d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:52.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.974+0000 7f4cc94ea640 1 -- 172.21.15.60:0/3481252601 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc406e740 con 0x7f4cc406f9f0 2024-05-09T05:00:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.975+0000 7f4cc94ea640 1 --2- 172.21.15.60:0/3481252601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cc406f9f0 0x7f4cc406ef40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc009dc0 tx=0x7f4cbc02a730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18849963f0d2ddb5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.975+0000 7f4cc37fe640 1 -- 172.21.15.60:0/3481252601 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cbc007740 con 0x7f4cc406f9f0 2024-05-09T05:00:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.975+0000 7f4cc37fe640 1 -- 172.21.15.60:0/3481252601 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cbc0078e0 con 0x7f4cc406f9f0 2024-05-09T05:00:52.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.975+0000 7f4cc37fe640 1 -- 172.21.15.60:0/3481252601 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cbc007cb0 con 0x7f4cc406f9f0 2024-05-09T05:00:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.976+0000 7f4ccaf4d640 1 -- 172.21.15.60:0/3481252601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cc406f9f0 msgr2=0x7f4cc406ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.976+0000 7f4ccaf4d640 1 --2- 172.21.15.60:0/3481252601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cc406f9f0 0x7f4cc406ef40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc009dc0 tx=0x7f4cbc02a730 comp rx=0 tx=0).stop 2024-05-09T05:00:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.976+0000 7f4ccaf4d640 1 -- 172.21.15.60:0/3481252601 shutdown_connections 2024-05-09T05:00:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.976+0000 7f4ccaf4d640 1 --2- 172.21.15.60:0/3481252601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cc406f9f0 0x7f4cc406ef40 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.976+0000 7f4ccaf4d640 1 -- 172.21.15.60:0/3481252601 >> 172.21.15.60:0/3481252601 conn(0x7f4cc409b850 msgr2=0x7f4cc409bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.976+0000 7f4ccaf4d640 1 -- 172.21.15.60:0/3481252601 shutdown_connections 2024-05-09T05:00:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.976+0000 7f4ccaf4d640 1 -- 172.21.15.60:0/3481252601 wait complete. 2024-05-09T05:00:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.977+0000 7f4ccaf4d640 1 Processor -- start 2024-05-09T05:00:52.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.977+0000 7f4ccaf4d640 1 -- start start 2024-05-09T05:00:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.978+0000 7f4ccaf4d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cc406f9f0 0x7f4cc415a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.978+0000 7f4ccaf4d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cc415f300 0x7f4cc40afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.978+0000 7f4ccaf4d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cc41606f0 0x7f4cc41615f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:52.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.978+0000 7f4ccaf4d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc40d3c80 con 0x7f4cc415f300 2024-05-09T05:00:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.978+0000 7f4ccaf4d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc40d3b00 con 0x7f4cc406f9f0 2024-05-09T05:00:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.978+0000 7f4ccaf4d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4cc406f480 con 0x7f4cc41606f0 2024-05-09T05:00:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.978+0000 7f4cc3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cc415f300 0x7f4cc40afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.978+0000 7f4cc3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cc415f300 0x7f4cc40afa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51054/0 (socket says 172.21.15.60:51054) 2024-05-09T05:00:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.978+0000 7f4cc3fff640 1 -- 172.21.15.60:0/1379603499 learned_addr learned my addr 172.21.15.60:0/1379603499 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.978+0000 7f4cc8ce9640 1 --2- 172.21.15.60:0/1379603499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cc406f9f0 0x7f4cc415a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.978+0000 7f4cc94ea640 1 --2- 172.21.15.60:0/1379603499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cc41606f0 0x7f4cc41615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.978+0000 7f4cc3fff640 1 -- 172.21.15.60:0/1379603499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cc41606f0 msgr2=0x7f4cc41615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.978+0000 7f4cc3fff640 1 --2- 172.21.15.60:0/1379603499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cc41606f0 0x7f4cc41615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.978+0000 7f4cc3fff640 1 -- 172.21.15.60:0/1379603499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cc406f9f0 msgr2=0x7f4cc415a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.978+0000 7f4cc3fff640 1 --2- 172.21.15.60:0/1379603499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cc406f9f0 0x7f4cc415a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.978+0000 7f4cc3fff640 1 -- 172.21.15.60:0/1379603499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc409e030 con 0x7f4cc415f300 2024-05-09T05:00:52.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.978+0000 7f4cc3fff640 1 --2- 172.21.15.60:0/1379603499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cc415f300 0x7f4cc40afa20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4cb000c410 tx=0x7f4cb000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.979+0000 7f4cc17fa640 1 -- 172.21.15.60:0/1379603499 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb0017070 con 0x7f4cc415f300 2024-05-09T05:00:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.979+0000 7f4cc17fa640 1 -- 172.21.15.60:0/1379603499 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cb0005ce0 con 0x7f4cc415f300 2024-05-09T05:00:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.979+0000 7f4ccaf4d640 1 -- 172.21.15.60:0/1379603499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cc409d270 con 0x7f4cc415f300 2024-05-09T05:00:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.979+0000 7f4cc17fa640 1 -- 172.21.15.60:0/1379603499 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb00072f0 con 0x7f4cc415f300 2024-05-09T05:00:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.979+0000 7f4ccaf4d640 1 -- 172.21.15.60:0/1379603499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cc409d830 con 0x7f4cc415f300 2024-05-09T05:00:52.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.980+0000 7f4ccaf4d640 1 -- 172.21.15.60:0/1379603499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cc4051040 con 0x7f4cc415f300 2024-05-09T05:00:52.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.980+0000 7f4cc17fa640 1 -- 172.21.15.60:0/1379603499 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4cb001a070 con 0x7f4cc415f300 2024-05-09T05:00:52.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.981+0000 7f4cc17fa640 1 --2- 172.21.15.60:0/1379603499 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4ca80762d0 0x7f4ca8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:52.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.981+0000 7f4cc17fa640 1 -- 172.21.15.60:0/1379603499 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4cb0096c10 con 0x7f4cc415f300 2024-05-09T05:00:52.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.981+0000 7f4cc8ce9640 1 --2- 172.21.15.60:0/1379603499 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4ca80762d0 0x7f4ca8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:52.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.982+0000 7f4cc8ce9640 1 --2- 172.21.15.60:0/1379603499 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4ca80762d0 0x7f4ca8078790 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8001810 tx=0x7f4cb8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:52.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:52.989+0000 7f4cc17fa640 1 -- 172.21.15.60:0/1379603499 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4cb005fca0 con 0x7f4cc415f300 2024-05-09T05:00:53.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.222+0000 7f4ccaf4d640 1 -- 172.21.15.60:0/1379603499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4cc415dbb0 con 0x7f4cc415f300 2024-05-09T05:00:53.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.222+0000 7f4cc17fa640 1 -- 172.21.15.60:0/1379603499 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7f4cb0063fa0 con 0x7f4cc415f300 2024-05-09T05:00:53.223 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:53.223 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1285340937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1285340937},{"type":"v1","addr":"172.21.15.60:6833","nonce":1285340937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2599193390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2599193390},{"type":"v1","addr":"172.21.15.114:6835","nonce":2599193390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3432609307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3432609307},{"type":"v1","addr":"172.21.15.114:6837","nonce":3432609307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:00:12.208556+0000","modified":"2024-05-09T05:00:31.347095+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2135423314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2135423314},{"type":"v1","addr":"172.21.15.60:6835","nonce":2135423314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-05-09T05:00:53.223 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2024-05-09T05:00:53.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.224+0000 7f4ccaf4d640 1 -- 172.21.15.60:0/1379603499 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4ca80762d0 msgr2=0x7f4ca8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:53.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.224+0000 7f4ccaf4d640 1 --2- 172.21.15.60:0/1379603499 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4ca80762d0 0x7f4ca8078790 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8001810 tx=0x7f4cb8009040 comp rx=0 tx=0).stop 2024-05-09T05:00:53.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.224+0000 7f4ccaf4d640 1 -- 172.21.15.60:0/1379603499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cc415f300 msgr2=0x7f4cc40afa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:53.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.224+0000 7f4ccaf4d640 1 --2- 172.21.15.60:0/1379603499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cc415f300 0x7f4cc40afa20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4cb000c410 tx=0x7f4cb000c8d0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.224+0000 7f4ccaf4d640 1 -- 172.21.15.60:0/1379603499 shutdown_connections 2024-05-09T05:00:53.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.224+0000 7f4ccaf4d640 1 --2- 172.21.15.60:0/1379603499 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4ca80762d0 0x7f4ca8078790 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.224+0000 7f4ccaf4d640 1 --2- 172.21.15.60:0/1379603499 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4cc41606f0 0x7f4cc41615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.224+0000 7f4ccaf4d640 1 --2- 172.21.15.60:0/1379603499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cc415f300 0x7f4cc40afa20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.224+0000 7f4ccaf4d640 1 --2- 172.21.15.60:0/1379603499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cc406f9f0 0x7f4cc415a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.224+0000 7f4ccaf4d640 1 -- 172.21.15.60:0/1379603499 >> 172.21.15.60:0/1379603499 conn(0x7f4cc409b850 msgr2=0x7f4cc415ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:53.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.225+0000 7f4ccaf4d640 1 -- 172.21.15.60:0/1379603499 shutdown_connections 2024-05-09T05:00:53.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.225+0000 7f4ccaf4d640 1 -- 172.21.15.60:0/1379603499 wait complete. 2024-05-09T05:00:53.236 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:00:53.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.341+0000 7f0871c45640 1 Processor -- start 2024-05-09T05:00:53.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.341+0000 7f0871c45640 1 -- start start 2024-05-09T05:00:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.342+0000 7f0871c45640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f086c15c3a0 0x7f086c070150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.342+0000 7f0871c45640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f086c0ac430 0x7f086c070690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.342+0000 7f0871c45640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f086c071b80 con 0x7f086c097de0 2024-05-09T05:00:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.342+0000 7f0871c45640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f086c071e60 con 0x7f086c15c3a0 2024-05-09T05:00:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.342+0000 7f0871c45640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f086c072140 con 0x7f086c0ac430 2024-05-09T05:00:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.342+0000 7f086b7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f086c15c3a0 0x7f086c070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.342+0000 7f086bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f086c097de0 0x7f086c099a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56466/0 (socket says 172.21.15.60:56466) 2024-05-09T05:00:53.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.342+0000 7f086bfff640 1 -- 172.21.15.60:0/1585954567 learned_addr learned my addr 172.21.15.60:0/1585954567 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.342+0000 7f086affd640 1 --2- 172.21.15.60:0/1585954567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f086c0ac430 0x7f086c070690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.342+0000 7f086b7fe640 1 -- 172.21.15.60:0/1585954567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f086c0ac430 msgr2=0x7f086c070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.342+0000 7f086b7fe640 1 --2- 172.21.15.60:0/1585954567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f086c0ac430 0x7f086c070690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.342+0000 7f086b7fe640 1 -- 172.21.15.60:0/1585954567 >> v1:172.21.15.60:6789/0 conn(0x7f086c097de0 legacy=0x7f086c099a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.342+0000 7f086b7fe640 1 -- 172.21.15.60:0/1585954567 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f086c072420 con 0x7f086c15c3a0 2024-05-09T05:00:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.342+0000 7f086a7fc640 1 -- 172.21.15.60:0/1585954567 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2418719155 0 0) 0x7f086c071b80 con 0x7f086c097de0 2024-05-09T05:00:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.343+0000 7f086b7fe640 1 --2- 172.21.15.60:0/1585954567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f086c15c3a0 0x7f086c070150 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f085c0097d0 tx=0x7f085c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a79410af357340d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.343+0000 7f086a7fc640 1 -- 172.21.15.60:0/1585954567 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c037070 con 0x7f086c15c3a0 2024-05-09T05:00:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.343+0000 7f086a7fc640 1 -- 172.21.15.60:0/1585954567 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f085c032440 con 0x7f086c15c3a0 2024-05-09T05:00:53.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.343+0000 7f086a7fc640 1 -- 172.21.15.60:0/1585954567 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c032810 con 0x7f086c15c3a0 2024-05-09T05:00:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.344+0000 7f0871c45640 1 -- 172.21.15.60:0/1585954567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f086c15c3a0 msgr2=0x7f086c070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.344+0000 7f0871c45640 1 --2- 172.21.15.60:0/1585954567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f086c15c3a0 0x7f086c070150 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f085c0097d0 tx=0x7f085c02a1c0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.344+0000 7f0871c45640 1 -- 172.21.15.60:0/1585954567 shutdown_connections 2024-05-09T05:00:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.344+0000 7f0871c45640 1 --2- 172.21.15.60:0/1585954567 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f086c0ac430 0x7f086c070690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.344+0000 7f0871c45640 1 --2- 172.21.15.60:0/1585954567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f086c15c3a0 0x7f086c070150 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.344+0000 7f0871c45640 1 -- 172.21.15.60:0/1585954567 >> 172.21.15.60:0/1585954567 conn(0x7f086c0972f0 msgr2=0x7f086c0976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.344+0000 7f0871c45640 1 -- 172.21.15.60:0/1585954567 shutdown_connections 2024-05-09T05:00:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.344+0000 7f0871c45640 1 -- 172.21.15.60:0/1585954567 wait complete. 2024-05-09T05:00:53.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.345+0000 7f0871c45640 1 Processor -- start 2024-05-09T05:00:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.345+0000 7f0871c45640 1 -- start start 2024-05-09T05:00:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.346+0000 7f0871c45640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f086c097de0 0x7f086c158f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.346+0000 7f0871c45640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f086c0ac430 0x7f086c158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.346+0000 7f0871c45640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f086c15c3a0 0x7f086c158840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.346+0000 7f0871c45640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f086c071810 con 0x7f086c097de0 2024-05-09T05:00:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.346+0000 7f0871c45640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f086c15c780 con 0x7f086c15c3a0 2024-05-09T05:00:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.346+0000 7f0871c45640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f086c073c30 con 0x7f086c0ac430 2024-05-09T05:00:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.346+0000 7f086b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f086c097de0 0x7f086c158f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.346+0000 7f086b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f086c097de0 0x7f086c158f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51060/0 (socket says 172.21.15.60:51060) 2024-05-09T05:00:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.346+0000 7f086b7fe640 1 -- 172.21.15.60:0/2782026752 learned_addr learned my addr 172.21.15.60:0/2782026752 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.346+0000 7f086bfff640 1 --2- 172.21.15.60:0/2782026752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f086c15c3a0 0x7f086c158840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:53.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.346+0000 7f086affd640 1 --2- 172.21.15.60:0/2782026752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f086c0ac430 0x7f086c158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.346+0000 7f086b7fe640 1 -- 172.21.15.60:0/2782026752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f086c0ac430 msgr2=0x7f086c158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.346+0000 7f086b7fe640 1 --2- 172.21.15.60:0/2782026752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f086c0ac430 0x7f086c158300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.346+0000 7f086b7fe640 1 -- 172.21.15.60:0/2782026752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f086c15c3a0 msgr2=0x7f086c158840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.346+0000 7f086b7fe640 1 --2- 172.21.15.60:0/2782026752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f086c15c3a0 0x7f086c158840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.347+0000 7f086b7fe640 1 -- 172.21.15.60:0/2782026752 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f086c168fd0 con 0x7f086c097de0 2024-05-09T05:00:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.347+0000 7f086bfff640 1 --2- 172.21.15.60:0/2782026752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f086c15c3a0 0x7f086c158840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.347+0000 7f086b7fe640 1 --2- 172.21.15.60:0/2782026752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f086c097de0 0x7f086c158f50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f085c035040 tx=0x7f085c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.347+0000 7f0853fff640 1 -- 172.21.15.60:0/2782026752 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c037050 con 0x7f086c097de0 2024-05-09T05:00:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.347+0000 7f0853fff640 1 -- 172.21.15.60:0/2782026752 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f085c038040 con 0x7f086c097de0 2024-05-09T05:00:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.347+0000 7f0871c45640 1 -- 172.21.15.60:0/2782026752 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f086c1692b0 con 0x7f086c097de0 2024-05-09T05:00:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.347+0000 7f0853fff640 1 -- 172.21.15.60:0/2782026752 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c004490 con 0x7f086c097de0 2024-05-09T05:00:53.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.347+0000 7f0871c45640 1 -- 172.21.15.60:0/2782026752 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f086c169870 con 0x7f086c097de0 2024-05-09T05:00:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.348+0000 7f0871c45640 1 -- 172.21.15.60:0/2782026752 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f086c051040 con 0x7f086c097de0 2024-05-09T05:00:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.349+0000 7f0853fff640 1 -- 172.21.15.60:0/2782026752 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f085c030900 con 0x7f086c097de0 2024-05-09T05:00:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.349+0000 7f0853fff640 1 --2- 172.21.15.60:0/2782026752 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0858076300 0x7f08580787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:53.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.349+0000 7f0853fff640 1 -- 172.21.15.60:0/2782026752 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f085c0b8330 con 0x7f086c097de0 2024-05-09T05:00:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.349+0000 7f086affd640 1 --2- 172.21.15.60:0/2782026752 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0858076300 0x7f08580787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.350+0000 7f086affd640 1 --2- 172.21.15.60:0/2782026752 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0858076300 0x7f08580787c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0860006fd0 tx=0x7f0860008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:53.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.353+0000 7f0853fff640 1 -- 172.21.15.60:0/2782026752 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f085c0819e0 con 0x7f086c097de0 2024-05-09T05:00:53.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.583+0000 7f0871c45640 1 -- 172.21.15.60:0/2782026752 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f086c150560 con 0x7f086c097de0 2024-05-09T05:00:53.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.583+0000 7f0853fff640 1 -- 172.21.15.60:0/2782026752 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7f085c0812e0 con 0x7f086c097de0 2024-05-09T05:00:53.584 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:53.584 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1285340937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1285340937},{"type":"v1","addr":"172.21.15.60:6833","nonce":1285340937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2599193390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2599193390},{"type":"v1","addr":"172.21.15.114:6835","nonce":2599193390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3432609307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":3432609307},{"type":"v1","addr":"172.21.15.114:6837","nonce":3432609307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:00:12.208556+0000","modified":"2024-05-09T05:00:31.347095+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4261":{"gid":4261,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2135423314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":2135423314},{"type":"v1","addr":"172.21.15.60:6835","nonce":2135423314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4261,"qdb_cluster":[4261]},"id":1}]} 2024-05-09T05:00:53.584 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2024-05-09T05:00:53.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.585+0000 7f0871c45640 1 -- 172.21.15.60:0/2782026752 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0858076300 msgr2=0x7f08580787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:53.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.585+0000 7f0871c45640 1 --2- 172.21.15.60:0/2782026752 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0858076300 0x7f08580787c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0860006fd0 tx=0x7f0860008040 comp rx=0 tx=0).stop 2024-05-09T05:00:53.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.585+0000 7f0871c45640 1 -- 172.21.15.60:0/2782026752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f086c097de0 msgr2=0x7f086c158f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:53.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.585+0000 7f0871c45640 1 --2- 172.21.15.60:0/2782026752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f086c097de0 0x7f086c158f50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f085c035040 tx=0x7f085c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.585+0000 7f0871c45640 1 -- 172.21.15.60:0/2782026752 shutdown_connections 2024-05-09T05:00:53.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.585+0000 7f0871c45640 1 --2- 172.21.15.60:0/2782026752 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0858076300 0x7f08580787c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.585+0000 7f0871c45640 1 --2- 172.21.15.60:0/2782026752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f086c15c3a0 0x7f086c158840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.585+0000 7f0871c45640 1 --2- 172.21.15.60:0/2782026752 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f086c0ac430 0x7f086c158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.585+0000 7f0871c45640 1 --2- 172.21.15.60:0/2782026752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f086c097de0 0x7f086c158f50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.585+0000 7f0871c45640 1 -- 172.21.15.60:0/2782026752 >> 172.21.15.60:0/2782026752 conn(0x7f086c0972f0 msgr2=0x7f086c15abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:53.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.585+0000 7f0871c45640 1 -- 172.21.15.60:0/2782026752 shutdown_connections 2024-05-09T05:00:53.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.585+0000 7f0871c45640 1 -- 172.21.15.60:0/2782026752 wait complete. 2024-05-09T05:00:53.597 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:00:53.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.701+0000 7f8a8cbb6640 1 Processor -- start 2024-05-09T05:00:53.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.701+0000 7f8a8cbb6640 1 -- start start 2024-05-09T05:00:53.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.702+0000 7f8a8cbb6640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8a8806fe30 0x7f8a88070290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.702+0000 7f8a8cbb6640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a88153360 con 0x7f8a880d0480 2024-05-09T05:00:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.702+0000 7f8a8cbb6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a8815c8b0 con 0x7f8a88152f80 2024-05-09T05:00:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.702+0000 7f8a8cbb6640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f8a8815cb90 con 0x7f8a8806fe30 2024-05-09T05:00:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.702+0000 7f8a86d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8a880d0480 0x7f8a880b10f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56482/0 (socket says 172.21.15.60:56482) 2024-05-09T05:00:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.702+0000 7f8a86d9d640 1 -- 172.21.15.60:0/2087635283 learned_addr learned my addr 172.21.15.60:0/2087635283 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.702+0000 7f8a85d9b640 1 --2- 172.21.15.60:0/2087635283 >> v2:172.21.15.114:3300/0 conn(0x7f8a8806fe30 0x7f8a88070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.703+0000 7f8a8559a640 1 -- 172.21.15.60:0/2087635283 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2947192331 0 0) 0x7f8a88153360 con 0x7f8a880d0480 2024-05-09T05:00:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.703+0000 7f8a8559a640 1 -- 172.21.15.60:0/2087635283 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a70003520 con 0x7f8a880d0480 2024-05-09T05:00:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.703+0000 7f8a8559a640 1 -- 172.21.15.60:0/2087635283 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3379956466 0 0) 0x7f8a8815c8b0 con 0x7f8a88152f80 2024-05-09T05:00:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.703+0000 7f8a8559a640 1 -- 172.21.15.60:0/2087635283 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a88153360 con 0x7f8a88152f80 2024-05-09T05:00:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.703+0000 7f8a85d9b640 1 -- 172.21.15.60:0/2087635283 >> v1:172.21.15.114:6789/0 conn(0x7f8a88152f80 legacy=0x7f8a880ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.703+0000 7f8a8559a640 1 -- 172.21.15.60:0/2087635283 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4230357860 0 0) 0x7f8a70003520 con 0x7f8a880d0480 2024-05-09T05:00:53.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.703+0000 7f8a85d9b640 1 -- 172.21.15.60:0/2087635283 >> v1:172.21.15.60:6789/0 conn(0x7f8a880d0480 legacy=0x7f8a880b10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:53.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.703+0000 7f8a85d9b640 1 -- 172.21.15.60:0/2087635283 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a8815ce70 con 0x7f8a8806fe30 2024-05-09T05:00:53.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.704+0000 7f8a85d9b640 1 --2- 172.21.15.60:0/2087635283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a8806fe30 0x7f8a88070290 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c0097a0 tx=0x7f8a7c02a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0f6025d4a55ee5c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:53.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.704+0000 7f8a8559a640 1 -- 172.21.15.60:0/2087635283 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c037070 con 0x7f8a8806fe30 2024-05-09T05:00:53.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.704+0000 7f8a8559a640 1 -- 172.21.15.60:0/2087635283 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a7c02abc0 con 0x7f8a8806fe30 2024-05-09T05:00:53.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.704+0000 7f8a8559a640 1 -- 172.21.15.60:0/2087635283 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a80001a90 con 0x7f8a8806fe30 2024-05-09T05:00:53.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.704+0000 7f8a8cbb6640 1 -- 172.21.15.60:0/2087635283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a8806fe30 msgr2=0x7f8a88070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:53.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.704+0000 7f8a8cbb6640 1 --2- 172.21.15.60:0/2087635283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a8806fe30 0x7f8a88070290 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c0097a0 tx=0x7f8a7c02a130 comp rx=0 tx=0).stop 2024-05-09T05:00:53.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.705+0000 7f8a8cbb6640 1 -- 172.21.15.60:0/2087635283 shutdown_connections 2024-05-09T05:00:53.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.705+0000 7f8a8cbb6640 1 --2- 172.21.15.60:0/2087635283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a8806fe30 0x7f8a88070290 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.705+0000 7f8a8cbb6640 1 -- 172.21.15.60:0/2087635283 >> 172.21.15.60:0/2087635283 conn(0x7f8a880b4850 msgr2=0x7f8a880b4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:53.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.705+0000 7f8a8cbb6640 1 -- 172.21.15.60:0/2087635283 shutdown_connections 2024-05-09T05:00:53.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.705+0000 7f8a8cbb6640 1 -- 172.21.15.60:0/2087635283 wait complete. 2024-05-09T05:00:53.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.706+0000 7f8a8cbb6640 1 Processor -- start 2024-05-09T05:00:53.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.706+0000 7f8a8cbb6640 1 -- start start 2024-05-09T05:00:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.707+0000 7f8a8cbb6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a8806fe30 0x7f8a881f6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.707+0000 7f8a8cbb6640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a880d0480 0x7f8a8814dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.707+0000 7f8a8cbb6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a88152f80 0x7f8a88150810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.707+0000 7f8a8cbb6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8815c670 con 0x7f8a88152f80 2024-05-09T05:00:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.707+0000 7f8a8cbb6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8815c4f0 con 0x7f8a8806fe30 2024-05-09T05:00:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.707+0000 7f8a8cbb6640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8a880b1800 con 0x7f8a880d0480 2024-05-09T05:00:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.707+0000 7f8a86d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a88152f80 0x7f8a88150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.707+0000 7f8a86d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a88152f80 0x7f8a88150810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51062/0 (socket says 172.21.15.60:51062) 2024-05-09T05:00:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.707+0000 7f8a86d9d640 1 -- 172.21.15.60:0/650375181 learned_addr learned my addr 172.21.15.60:0/650375181 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.707+0000 7f8a8659c640 1 --2- 172.21.15.60:0/650375181 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a8806fe30 0x7f8a881f6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.707+0000 7f8a85d9b640 1 --2- 172.21.15.60:0/650375181 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a880d0480 0x7f8a8814dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:53.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.707+0000 7f8a86d9d640 1 -- 172.21.15.60:0/650375181 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a880d0480 msgr2=0x7f8a8814dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:53.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.707+0000 7f8a86d9d640 1 --2- 172.21.15.60:0/650375181 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a880d0480 0x7f8a8814dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.707+0000 7f8a86d9d640 1 -- 172.21.15.60:0/650375181 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a8806fe30 msgr2=0x7f8a881f6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:53.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.707+0000 7f8a86d9d640 1 --2- 172.21.15.60:0/650375181 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a8806fe30 0x7f8a881f6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.707+0000 7f8a86d9d640 1 -- 172.21.15.60:0/650375181 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a8813c180 con 0x7f8a88152f80 2024-05-09T05:00:53.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.707+0000 7f8a86d9d640 1 --2- 172.21.15.60:0/650375181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a88152f80 0x7f8a88150810 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8a80007fd0 tx=0x7f8a8000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:53.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.707+0000 7f8a76ffd640 1 -- 172.21.15.60:0/650375181 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a80018070 con 0x7f8a88152f80 2024-05-09T05:00:53.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.708+0000 7f8a76ffd640 1 -- 172.21.15.60:0/650375181 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a8000a640 con 0x7f8a88152f80 2024-05-09T05:00:53.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.708+0000 7f8a8cbb6640 1 -- 172.21.15.60:0/650375181 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a8813c460 con 0x7f8a88152f80 2024-05-09T05:00:53.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.708+0000 7f8a76ffd640 1 -- 172.21.15.60:0/650375181 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8000aab0 con 0x7f8a88152f80 2024-05-09T05:00:53.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.708+0000 7f8a8cbb6640 1 -- 172.21.15.60:0/650375181 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a8813ca20 con 0x7f8a88152f80 2024-05-09T05:00:53.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.709+0000 7f8a8cbb6640 1 -- 172.21.15.60:0/650375181 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a880b24b0 con 0x7f8a88152f80 2024-05-09T05:00:53.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.709+0000 7f8a76ffd640 1 -- 172.21.15.60:0/650375181 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8a8001b070 con 0x7f8a88152f80 2024-05-09T05:00:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.710+0000 7f8a76ffd640 1 --2- 172.21.15.60:0/650375181 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8a680762d0 0x7f8a68078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.710+0000 7f8a76ffd640 1 -- 172.21.15.60:0/650375181 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f8a80096fc0 con 0x7f8a88152f80 2024-05-09T05:00:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.710+0000 7f8a8659c640 1 --2- 172.21.15.60:0/650375181 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8a680762d0 0x7f8a68078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.711+0000 7f8a8659c640 1 --2- 172.21.15.60:0/650375181 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8a680762d0 0x7f8a68078790 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8a78001120 tx=0x7f8a78009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:53.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.714+0000 7f8a76ffd640 1 -- 172.21.15.60:0/650375181 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8a880b24b0 con 0x7f8a88152f80 2024-05-09T05:00:53.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.907+0000 7f8a8cbb6640 1 -- 172.21.15.60:0/650375181 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a88051040 con 0x7f8a88152f80 2024-05-09T05:00:53.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.908+0000 7f8a76ffd640 1 -- 172.21.15.60:0/650375181 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18490 (secure 0 0 0) 0x7f8a8005ff90 con 0x7f8a88152f80 2024-05-09T05:00:53.909 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:53.909 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:00:17.248958+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:00:10.358133+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:00:11.157769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T05:00:14.850459+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:00:53.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.910+0000 7f8a8cbb6640 1 -- 172.21.15.60:0/650375181 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8a680762d0 msgr2=0x7f8a68078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:53.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.910+0000 7f8a8cbb6640 1 --2- 172.21.15.60:0/650375181 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8a680762d0 0x7f8a68078790 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8a78001120 tx=0x7f8a78009040 comp rx=0 tx=0).stop 2024-05-09T05:00:53.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.910+0000 7f8a8cbb6640 1 -- 172.21.15.60:0/650375181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a88152f80 msgr2=0x7f8a88150810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:53.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.910+0000 7f8a8cbb6640 1 --2- 172.21.15.60:0/650375181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a88152f80 0x7f8a88150810 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8a80007fd0 tx=0x7f8a8000ccd0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.911+0000 7f8a8cbb6640 1 -- 172.21.15.60:0/650375181 shutdown_connections 2024-05-09T05:00:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.911+0000 7f8a8cbb6640 1 --2- 172.21.15.60:0/650375181 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8a680762d0 0x7f8a68078790 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.911+0000 7f8a8cbb6640 1 --2- 172.21.15.60:0/650375181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a88152f80 0x7f8a88150810 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.911+0000 7f8a8cbb6640 1 --2- 172.21.15.60:0/650375181 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a880d0480 0x7f8a8814dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.911+0000 7f8a8cbb6640 1 --2- 172.21.15.60:0/650375181 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a8806fe30 0x7f8a881f6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.911+0000 7f8a8cbb6640 1 -- 172.21.15.60:0/650375181 >> 172.21.15.60:0/650375181 conn(0x7f8a880b4850 msgr2=0x7f8a88072130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.911+0000 7f8a8cbb6640 1 -- 172.21.15.60:0/650375181 shutdown_connections 2024-05-09T05:00:53.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:53.911+0000 7f8a8cbb6640 1 -- 172.21.15.60:0/650375181 wait complete. 2024-05-09T05:00:53.923 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T05:00:54.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.027+0000 7fab3bbd4640 1 Processor -- start 2024-05-09T05:00:54.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.028+0000 7fab3bbd4640 1 -- start start 2024-05-09T05:00:54.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.028+0000 7fab3bbd4640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fab341598f0 0x7fab34160ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:54.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.028+0000 7fab3bbd4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab341605d0 0x7fab340b1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:54.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.028+0000 7fab3bbd4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fab34072130 0x7fab34072570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:54.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.028+0000 7fab3bbd4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fab340addd0 con 0x7fab34072130 2024-05-09T05:00:54.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.028+0000 7fab3bbd4640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fab34072ab0 con 0x7fab341598f0 2024-05-09T05:00:54.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.028+0000 7fab3bbd4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fab340b2500 con 0x7fab341605d0 2024-05-09T05:00:54.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.029+0000 7fab3a171640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fab34072130 0x7fab34072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:54.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.029+0000 7fab3a171640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fab34072130 0x7fab34072570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51072/0 (socket says 172.21.15.60:51072) 2024-05-09T05:00:54.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.029+0000 7fab3a171640 1 -- 172.21.15.60:0/1722381585 learned_addr learned my addr 172.21.15.60:0/1722381585 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:54.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.029+0000 7fab39970640 1 --2- 172.21.15.60:0/1722381585 >> v2:172.21.15.114:3300/0 conn(0x7fab341598f0 0x7fab34160ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:54.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.029+0000 7fab3a171640 1 -- 172.21.15.60:0/1722381585 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab341605d0 msgr2=0x7fab340b1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:54.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.029+0000 7fab3a171640 1 --2- 172.21.15.60:0/1722381585 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab341605d0 0x7fab340b1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.029+0000 7fab3a171640 1 -- 172.21.15.60:0/1722381585 >> v2:172.21.15.114:3300/0 conn(0x7fab341598f0 msgr2=0x7fab34160ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:54.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.029+0000 7fab3a171640 1 --2- 172.21.15.60:0/1722381585 >> v2:172.21.15.114:3300/0 conn(0x7fab341598f0 0x7fab34160ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.029+0000 7fab3a171640 1 -- 172.21.15.60:0/1722381585 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab340b27e0 con 0x7fab34072130 2024-05-09T05:00:54.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.029+0000 7fab3a171640 1 --2- 172.21.15.60:0/1722381585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab34072130 0x7fab34072570 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fab34070dd0 tx=0x7fab2c02a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd3cc0540a299ff server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:54.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.030+0000 7fab3896e640 1 -- 172.21.15.60:0/1722381585 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c037070 con 0x7fab34072130 2024-05-09T05:00:54.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.030+0000 7fab3896e640 1 -- 172.21.15.60:0/1722381585 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab2c02ae00 con 0x7fab34072130 2024-05-09T05:00:54.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.030+0000 7fab3896e640 1 -- 172.21.15.60:0/1722381585 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c032660 con 0x7fab34072130 2024-05-09T05:00:54.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.030+0000 7fab3bbd4640 1 -- 172.21.15.60:0/1722381585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab34072130 msgr2=0x7fab34072570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:54.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.030+0000 7fab3bbd4640 1 --2- 172.21.15.60:0/1722381585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab34072130 0x7fab34072570 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fab34070dd0 tx=0x7fab2c02a0b0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.030+0000 7fab3bbd4640 1 -- 172.21.15.60:0/1722381585 shutdown_connections 2024-05-09T05:00:54.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.030+0000 7fab3bbd4640 1 --2- 172.21.15.60:0/1722381585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab34072130 0x7fab34072570 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.030+0000 7fab3bbd4640 1 --2- 172.21.15.60:0/1722381585 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab341605d0 0x7fab340b1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.030+0000 7fab3bbd4640 1 --2- 172.21.15.60:0/1722381585 >> v2:172.21.15.114:3300/0 conn(0x7fab341598f0 0x7fab34160ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.030+0000 7fab3bbd4640 1 -- 172.21.15.60:0/1722381585 >> 172.21.15.60:0/1722381585 conn(0x7fab340b4850 msgr2=0x7fab340b4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:54.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.030+0000 7fab3bbd4640 1 -- 172.21.15.60:0/1722381585 shutdown_connections 2024-05-09T05:00:54.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.030+0000 7fab3bbd4640 1 -- 172.21.15.60:0/1722381585 wait complete. 2024-05-09T05:00:54.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.032+0000 7fab3bbd4640 1 Processor -- start 2024-05-09T05:00:54.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.032+0000 7fab3bbd4640 1 -- start start 2024-05-09T05:00:54.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.032+0000 7fab3bbd4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab34072130 0x7fab3415c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:54.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.032+0000 7fab3bbd4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab341598f0 0x7fab340aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:54.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.032+0000 7fab3bbd4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab341605d0 0x7fab3414a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:54.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.032+0000 7fab3bbd4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fab34161020 con 0x7fab341605d0 2024-05-09T05:00:54.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.032+0000 7fab3bbd4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fab340ae020 con 0x7fab34072130 2024-05-09T05:00:54.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.032+0000 7fab3bbd4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fab340b21f0 con 0x7fab341598f0 2024-05-09T05:00:54.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.032+0000 7fab3a171640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab341605d0 0x7fab3414a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:54.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.032+0000 7fab3a171640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab341605d0 0x7fab3414a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51084/0 (socket says 172.21.15.60:51084) 2024-05-09T05:00:54.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.032+0000 7fab3a171640 1 -- 172.21.15.60:0/2073409143 learned_addr learned my addr 172.21.15.60:0/2073409143 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:54.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.033+0000 7fab39970640 1 --2- 172.21.15.60:0/2073409143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab34072130 0x7fab3415c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:54.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.033+0000 7fab3916f640 1 --2- 172.21.15.60:0/2073409143 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab341598f0 0x7fab340aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:54.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.033+0000 7fab3a171640 1 -- 172.21.15.60:0/2073409143 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab341598f0 msgr2=0x7fab340aa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:54.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.033+0000 7fab3a171640 1 --2- 172.21.15.60:0/2073409143 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab341598f0 0x7fab340aa4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.033+0000 7fab3a171640 1 -- 172.21.15.60:0/2073409143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab34072130 msgr2=0x7fab3415c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:54.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.033+0000 7fab3a171640 1 --2- 172.21.15.60:0/2073409143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab34072130 0x7fab3415c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.033+0000 7fab3a171640 1 -- 172.21.15.60:0/2073409143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab3415b440 con 0x7fab341605d0 2024-05-09T05:00:54.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.033+0000 7fab3916f640 1 --2- 172.21.15.60:0/2073409143 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab341598f0 0x7fab340aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:54.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.033+0000 7fab3a171640 1 --2- 172.21.15.60:0/2073409143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab341605d0 0x7fab3414a730 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fab2c0090b0 tx=0x7fab2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:54.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.033+0000 7fab2a7fc640 1 -- 172.21.15.60:0/2073409143 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c041070 con 0x7fab341605d0 2024-05-09T05:00:54.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.033+0000 7fab2a7fc640 1 -- 172.21.15.60:0/2073409143 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab2c03c670 con 0x7fab341605d0 2024-05-09T05:00:54.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.033+0000 7fab2a7fc640 1 -- 172.21.15.60:0/2073409143 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c037040 con 0x7fab341605d0 2024-05-09T05:00:54.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.033+0000 7fab3bbd4640 1 -- 172.21.15.60:0/2073409143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab3415b720 con 0x7fab341605d0 2024-05-09T05:00:54.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.033+0000 7fab3bbd4640 1 -- 172.21.15.60:0/2073409143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab3415bce0 con 0x7fab341605d0 2024-05-09T05:00:54.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.034+0000 7fab3bbd4640 1 -- 172.21.15.60:0/2073409143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab34051040 con 0x7fab341605d0 2024-05-09T05:00:54.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.035+0000 7fab2a7fc640 1 -- 172.21.15.60:0/2073409143 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fab2c03a070 con 0x7fab341605d0 2024-05-09T05:00:54.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.035+0000 7fab2a7fc640 1 --2- 172.21.15.60:0/2073409143 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fab180762d0 0x7fab18078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:54.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.035+0000 7fab2a7fc640 1 -- 172.21.15.60:0/2073409143 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fab2c0b7ad0 con 0x7fab341605d0 2024-05-09T05:00:54.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.036+0000 7fab39970640 1 --2- 172.21.15.60:0/2073409143 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fab180762d0 0x7fab18078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:54.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.036+0000 7fab39970640 1 --2- 172.21.15.60:0/2073409143 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fab180762d0 0x7fab18078790 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fab34070ba0 tx=0x7fab300039b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:54.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.039+0000 7fab2a7fc640 1 -- 172.21.15.60:0/2073409143 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fab2c081180 con 0x7fab341605d0 2024-05-09T05:00:54.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.273+0000 7fab3bbd4640 1 -- 172.21.15.60:0/2073409143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fab340694a0 con 0x7fab341605d0 2024-05-09T05:00:54.276 INFO:tasks.ceph.mon.a.smithi060.stderr:2024-05-09T05:00:54.273+0000 7fde6feeb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T05:00:54.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.274+0000 7fab2a7fc640 1 -- 172.21.15.60:0/2073409143 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) ==== 161+0+0 (secure 0 0 0) 0x7fab2c089170 con 0x7fab341605d0 2024-05-09T05:00:54.276 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T05:00:54.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.277+0000 7fab3bbd4640 1 -- 172.21.15.60:0/2073409143 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fab180762d0 msgr2=0x7fab18078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:54.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.277+0000 7fab3bbd4640 1 --2- 172.21.15.60:0/2073409143 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fab180762d0 0x7fab18078790 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fab34070ba0 tx=0x7fab300039b0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.277+0000 7fab3bbd4640 1 -- 172.21.15.60:0/2073409143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab341605d0 msgr2=0x7fab3414a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:54.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.277+0000 7fab3bbd4640 1 --2- 172.21.15.60:0/2073409143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab341605d0 0x7fab3414a730 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fab2c0090b0 tx=0x7fab2c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.277+0000 7fab3bbd4640 1 -- 172.21.15.60:0/2073409143 shutdown_connections 2024-05-09T05:00:54.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.277+0000 7fab3bbd4640 1 --2- 172.21.15.60:0/2073409143 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fab180762d0 0x7fab18078790 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.277+0000 7fab3bbd4640 1 --2- 172.21.15.60:0/2073409143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab341605d0 0x7fab3414a730 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.277+0000 7fab3bbd4640 1 --2- 172.21.15.60:0/2073409143 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fab341598f0 0x7fab340aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.277+0000 7fab3bbd4640 1 --2- 172.21.15.60:0/2073409143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab34072130 0x7fab3415c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.277+0000 7fab3bbd4640 1 -- 172.21.15.60:0/2073409143 >> 172.21.15.60:0/2073409143 conn(0x7fab340b4850 msgr2=0x7fab340b4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:54.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.277+0000 7fab3bbd4640 1 -- 172.21.15.60:0/2073409143 shutdown_connections 2024-05-09T05:00:54.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.277+0000 7fab3bbd4640 1 -- 172.21.15.60:0/2073409143 wait complete. 2024-05-09T05:00:54.280 INFO:tasks.ceph.mds.c.smithi060.stderr: -4139> 2024-05-09T05:00:18.075+0000 7fc7ec960640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T05:00:54.289 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-09T05:00:54.310 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2024-05-09T05:00:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.398+0000 7f6347809640 1 Processor -- start 2024-05-09T05:00:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.399+0000 7f6347809640 1 -- start start 2024-05-09T05:00:54.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.399+0000 7f6347809640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6340072130 0x7f63400725b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:54.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.399+0000 7f6347809640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f63400adbb0 0x7f63400ae020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:54.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.399+0000 7f6347809640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6340159d10 con 0x7f6340159930 2024-05-09T05:00:54.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.399+0000 7f6347809640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f634006f8e0 con 0x7f6340072130 2024-05-09T05:00:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.399+0000 7f6347809640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f634006fbc0 con 0x7f63400adbb0 2024-05-09T05:00:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.399+0000 7f6344da4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6340072130 0x7f63400725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.399+0000 7f63455a5640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6340159930 0x7f63401601b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56488/0 (socket says 172.21.15.60:56488) 2024-05-09T05:00:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.400+0000 7f63455a5640 1 -- 172.21.15.60:0/738964873 learned_addr learned my addr 172.21.15.60:0/738964873 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.400+0000 7f6345da6640 1 --2- 172.21.15.60:0/738964873 >> v2:172.21.15.114:3300/0 conn(0x7f63400adbb0 0x7f63400ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.400+0000 7f6344da4640 1 -- 172.21.15.60:0/738964873 >> v2:172.21.15.114:3300/0 conn(0x7f63400adbb0 msgr2=0x7f63400ae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.400+0000 7f6344da4640 1 --2- 172.21.15.60:0/738964873 >> v2:172.21.15.114:3300/0 conn(0x7f63400adbb0 0x7f63400ae020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.400+0000 7f6344da4640 1 -- 172.21.15.60:0/738964873 >> v1:172.21.15.60:6789/0 conn(0x7f6340159930 legacy=0x7f63401601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.400+0000 7f6337fff640 1 -- 172.21.15.60:0/738964873 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3415342110 0 0) 0x7f633c001110 con 0x7f6340159930 2024-05-09T05:00:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.400+0000 7f6344da4640 1 -- 172.21.15.60:0/738964873 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f634006fea0 con 0x7f6340072130 2024-05-09T05:00:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.401+0000 7f6344da4640 1 --2- 172.21.15.60:0/738964873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6340072130 0x7f63400725b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f634006db90 tx=0x7f633002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1366adf9a6b54c2c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.401+0000 7f6337fff640 1 -- 172.21.15.60:0/738964873 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6330037070 con 0x7f6340072130 2024-05-09T05:00:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.401+0000 7f6337fff640 1 -- 172.21.15.60:0/738964873 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6330032440 con 0x7f6340072130 2024-05-09T05:00:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.401+0000 7f6337fff640 1 -- 172.21.15.60:0/738964873 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6330032810 con 0x7f6340072130 2024-05-09T05:00:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.401+0000 7f6347809640 1 -- 172.21.15.60:0/738964873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6340072130 msgr2=0x7f63400725b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.401+0000 7f6347809640 1 --2- 172.21.15.60:0/738964873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6340072130 0x7f63400725b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f634006db90 tx=0x7f633002a1a0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.401+0000 7f6347809640 1 -- 172.21.15.60:0/738964873 shutdown_connections 2024-05-09T05:00:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.401+0000 7f6347809640 1 --2- 172.21.15.60:0/738964873 >> v2:172.21.15.114:3300/0 conn(0x7f63400adbb0 0x7f63400ae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.401+0000 7f6347809640 1 --2- 172.21.15.60:0/738964873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6340072130 0x7f63400725b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.401+0000 7f6347809640 1 -- 172.21.15.60:0/738964873 >> 172.21.15.60:0/738964873 conn(0x7f63400b4c30 msgr2=0x7f6340073240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.401+0000 7f6347809640 1 -- 172.21.15.60:0/738964873 shutdown_connections 2024-05-09T05:00:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.401+0000 7f6347809640 1 -- 172.21.15.60:0/738964873 wait complete. 2024-05-09T05:00:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.403+0000 7f6347809640 1 Processor -- start 2024-05-09T05:00:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.403+0000 7f6347809640 1 -- start start 2024-05-09T05:00:54.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.403+0000 7f6347809640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6340072130 0x7f634015b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:54.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.403+0000 7f6347809640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63400adbb0 0x7f634006c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:54.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.403+0000 7f6347809640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6340159930 0x7f634006cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:54.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.403+0000 7f6347809640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f634015fae0 con 0x7f6340159930 2024-05-09T05:00:54.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.403+0000 7f6347809640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6340070460 con 0x7f63400adbb0 2024-05-09T05:00:54.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.403+0000 7f6347809640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6340145b10 con 0x7f6340072130 2024-05-09T05:00:54.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.403+0000 7f6345da6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6340159930 0x7f634006cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:54.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.403+0000 7f6345da6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6340159930 0x7f634006cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51120/0 (socket says 172.21.15.60:51120) 2024-05-09T05:00:54.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.403+0000 7f6345da6640 1 -- 172.21.15.60:0/329700834 learned_addr learned my addr 172.21.15.60:0/329700834 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:54.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.404+0000 7f6344da4640 1 --2- 172.21.15.60:0/329700834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63400adbb0 0x7f634006c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:54.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.404+0000 7f6345da6640 1 -- 172.21.15.60:0/329700834 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6340072130 msgr2=0x7f634015b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:54.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.404+0000 7f6345da6640 1 --2- 172.21.15.60:0/329700834 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6340072130 0x7f634015b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.404+0000 7f6345da6640 1 -- 172.21.15.60:0/329700834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63400adbb0 msgr2=0x7f634006c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:54.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.404+0000 7f6345da6640 1 --2- 172.21.15.60:0/329700834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63400adbb0 0x7f634006c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:54.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.404+0000 7f6345da6640 1 -- 172.21.15.60:0/329700834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f634015c3d0 con 0x7f6340159930 2024-05-09T05:00:54.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.404+0000 7f6345da6640 1 --2- 172.21.15.60:0/329700834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6340159930 0x7f634006cc10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f633800cd70 tx=0x7f633800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:54.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.404+0000 7f6335ffb640 1 -- 172.21.15.60:0/329700834 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6338003b90 con 0x7f6340159930 2024-05-09T05:00:54.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.404+0000 7f6335ffb640 1 -- 172.21.15.60:0/329700834 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6338003d30 con 0x7f6340159930 2024-05-09T05:00:54.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.404+0000 7f6347809640 1 -- 172.21.15.60:0/329700834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f634015c6b0 con 0x7f6340159930 2024-05-09T05:00:54.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.404+0000 7f6335ffb640 1 -- 172.21.15.60:0/329700834 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6338004240 con 0x7f6340159930 2024-05-09T05:00:54.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.404+0000 7f6347809640 1 -- 172.21.15.60:0/329700834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6340145d60 con 0x7f6340159930 2024-05-09T05:00:54.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.405+0000 7f6347809640 1 -- 172.21.15.60:0/329700834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6340070740 con 0x7f6340159930 2024-05-09T05:00:54.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.406+0000 7f6335ffb640 1 -- 172.21.15.60:0/329700834 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f63380043e0 con 0x7f6340159930 2024-05-09T05:00:54.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.406+0000 7f6335ffb640 1 --2- 172.21.15.60:0/329700834 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f63240762d0 0x7f6324078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:54.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.406+0000 7f6335ffb640 1 -- 172.21.15.60:0/329700834 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6618+0+0 (secure 0 0 0) 0x7f63380983f0 con 0x7f6340159930 2024-05-09T05:00:54.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.407+0000 7f63455a5640 1 --2- 172.21.15.60:0/329700834 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f63240762d0 0x7f6324078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:54.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.407+0000 7f63455a5640 1 --2- 172.21.15.60:0/329700834 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f63240762d0 0x7f6324078790 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f633c007fd0 tx=0x7f633c007960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:54.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.411+0000 7f6335ffb640 1 -- 172.21.15.60:0/329700834 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6340070740 con 0x7f6340159930 2024-05-09T05:00:54.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:54.639+0000 7f6347809640 1 -- 172.21.15.60:0/329700834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f6340145f60 con 0x7f6340159930 2024-05-09T05:00:55.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.279+0000 7f6335ffb640 1 -- 172.21.15.60:0/329700834 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f6340145f60 con 0x7f6340159930 2024-05-09T05:00:55.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.282+0000 7f6347809640 1 -- 172.21.15.60:0/329700834 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f63240762d0 msgr2=0x7f6324078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:55.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.282+0000 7f6347809640 1 --2- 172.21.15.60:0/329700834 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f63240762d0 0x7f6324078790 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f633c007fd0 tx=0x7f633c007960 comp rx=0 tx=0).stop 2024-05-09T05:00:55.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.282+0000 7f6347809640 1 -- 172.21.15.60:0/329700834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6340159930 msgr2=0x7f634006cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:55.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.282+0000 7f6347809640 1 --2- 172.21.15.60:0/329700834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6340159930 0x7f634006cc10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f633800cd70 tx=0x7f633800a660 comp rx=0 tx=0).stop 2024-05-09T05:00:55.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.282+0000 7f6347809640 1 -- 172.21.15.60:0/329700834 shutdown_connections 2024-05-09T05:00:55.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.282+0000 7f6347809640 1 --2- 172.21.15.60:0/329700834 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f63240762d0 0x7f6324078790 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:55.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.282+0000 7f6347809640 1 --2- 172.21.15.60:0/329700834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6340159930 0x7f634006cc10 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:55.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.282+0000 7f6347809640 1 --2- 172.21.15.60:0/329700834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63400adbb0 0x7f634006c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:55.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.282+0000 7f6347809640 1 --2- 172.21.15.60:0/329700834 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6340072130 0x7f634015b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:55.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.282+0000 7f6347809640 1 -- 172.21.15.60:0/329700834 >> 172.21.15.60:0/329700834 conn(0x7f63400b4c30 msgr2=0x7f6340161450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:55.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.282+0000 7f6347809640 1 -- 172.21.15.60:0/329700834 shutdown_connections 2024-05-09T05:00:55.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.282+0000 7f6347809640 1 -- 172.21.15.60:0/329700834 wait complete. 2024-05-09T05:00:55.295 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-09T05:00:55.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.446+0000 7f79723da640 1 Processor -- start 2024-05-09T05:00:55.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.446+0000 7f79723da640 1 -- start start 2024-05-09T05:00:55.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.447+0000 7f79723da640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f796c0d4cc0 0x7f796c0b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:55.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.447+0000 7f79723da640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f796c0b5a90 con 0x7f796c1481b0 2024-05-09T05:00:55.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.447+0000 7f79723da640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f796c0b5d70 con 0x7f796c148710 2024-05-09T05:00:55.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.447+0000 7f79723da640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f796c0b6050 con 0x7f796c0d4cc0 2024-05-09T05:00:55.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.447+0000 7f796bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f796c1481b0 0x7f796c0d45b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56496/0 (socket says 172.21.15.60:56496) 2024-05-09T05:00:55.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.447+0000 7f796bfff640 1 -- 172.21.15.60:0/378325744 learned_addr learned my addr 172.21.15.60:0/378325744 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.447+0000 7f796b7fe640 1 -- 172.21.15.60:0/378325744 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3805273139 0 0) 0x7f796c0b5a90 con 0x7f796c1481b0 2024-05-09T05:00:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f796b7fe640 1 -- 172.21.15.60:0/378325744 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7958003520 con 0x7f796c1481b0 2024-05-09T05:00:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f7963fff640 1 --2- 172.21.15.60:0/378325744 >> v2:172.21.15.114:3300/0 conn(0x7f796c0d4cc0 0x7f796c0b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f796b7fe640 1 -- 172.21.15.60:0/378325744 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2832445762 0 0) 0x7f7958003520 con 0x7f796c1481b0 2024-05-09T05:00:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f796b7fe640 1 -- 172.21.15.60:0/378325744 >> v2:172.21.15.114:3300/0 conn(0x7f796c0d4cc0 msgr2=0x7f796c0b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f796b7fe640 1 --2- 172.21.15.60:0/378325744 >> v2:172.21.15.114:3300/0 conn(0x7f796c0d4cc0 0x7f796c0b54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f796b7fe640 1 -- 172.21.15.60:0/378325744 >> v1:172.21.15.114:6789/0 conn(0x7f796c148710 legacy=0x7f796c0d1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f796b7fe640 1 -- 172.21.15.60:0/378325744 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f796c0b6330 con 0x7f796c1481b0 2024-05-09T05:00:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f796b7fe640 1 -- 172.21.15.60:0/378325744 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f795c0016f0 con 0x7f796c1481b0 2024-05-09T05:00:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f796b7fe640 1 -- 172.21.15.60:0/378325744 >> v1:172.21.15.60:6789/0 conn(0x7f796c1481b0 legacy=0x7f796c0d45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f796b7fe640 1 --2- 172.21.15.60:0/378325744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7958003a50 0x7f7958003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f796b7fe640 1 --2- 172.21.15.60:0/378325744 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7958004800 0x7f7958004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f796b7fe640 1 --2- 172.21.15.60:0/378325744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79580052e0 0x7f795800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f796b7fe640 1 -- 172.21.15.60:0/378325744 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f796c0b6050 con 0x7f79580052e0 2024-05-09T05:00:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f796b7fe640 1 -- 172.21.15.60:0/378325744 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f795800c1b0 con 0x7f7958003a50 2024-05-09T05:00:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f796b7fe640 1 -- 172.21.15.60:0/378325744 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f795800c3b0 con 0x7f7958004800 2024-05-09T05:00:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f7970977640 1 --2- 172.21.15.60:0/378325744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79580052e0 0x7f795800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f796bfff640 1 --2- 172.21.15.60:0/378325744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7958003a50 0x7f7958003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f7963fff640 1 --2- 172.21.15.60:0/378325744 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7958004800 0x7f7958004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f7970977640 1 -- 172.21.15.60:0/378325744 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7958004800 msgr2=0x7f7958004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f7970977640 1 --2- 172.21.15.60:0/378325744 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7958004800 0x7f7958004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f7970977640 1 -- 172.21.15.60:0/378325744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7958003a50 msgr2=0x7f7958003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f7970977640 1 --2- 172.21.15.60:0/378325744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7958003a50 0x7f7958003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:55.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.448+0000 7f7970977640 1 -- 172.21.15.60:0/378325744 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7958003520 con 0x7f79580052e0 2024-05-09T05:00:55.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.449+0000 7f7970977640 1 -- 172.21.15.60:0/378325744 reap_dead start 2024-05-09T05:00:55.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.449+0000 7f7970977640 1 --2- 172.21.15.60:0/378325744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79580052e0 0x7f795800b9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7964001e60 tx=0x7f7958005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3bb8277c7fa98c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:55.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.449+0000 7f796b7fe640 1 -- 172.21.15.60:0/378325744 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7964001ab0 con 0x7f79580052e0 2024-05-09T05:00:55.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.449+0000 7f796b7fe640 1 -- 172.21.15.60:0/378325744 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f796402ae40 con 0x7f79580052e0 2024-05-09T05:00:55.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.449+0000 7f796b7fe640 1 -- 172.21.15.60:0/378325744 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7964003b70 con 0x7f79580052e0 2024-05-09T05:00:55.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.450+0000 7f79723da640 1 -- 172.21.15.60:0/378325744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79580052e0 msgr2=0x7f795800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:55.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.450+0000 7f79723da640 1 --2- 172.21.15.60:0/378325744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79580052e0 0x7f795800b9c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7964001e60 tx=0x7f7958005830 comp rx=0 tx=0).stop 2024-05-09T05:00:55.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.450+0000 7f79723da640 1 -- 172.21.15.60:0/378325744 shutdown_connections 2024-05-09T05:00:55.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.450+0000 7f79723da640 1 --2- 172.21.15.60:0/378325744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79580052e0 0x7f795800b9c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:55.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.450+0000 7f79723da640 1 -- 172.21.15.60:0/378325744 >> 172.21.15.60:0/378325744 conn(0x7f796c06f640 msgr2=0x7f796c073840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:55.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.450+0000 7f79723da640 1 -- 172.21.15.60:0/378325744 shutdown_connections 2024-05-09T05:00:55.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.450+0000 7f79723da640 1 -- 172.21.15.60:0/378325744 wait complete. 2024-05-09T05:00:55.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.451+0000 7f79723da640 1 Processor -- start 2024-05-09T05:00:55.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.451+0000 7f79723da640 1 -- start start 2024-05-09T05:00:55.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.452+0000 7f79723da640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79580052e0 0x7f796c13c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:55.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.452+0000 7f79723da640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f796c1621d0 0x7f796c13b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:55.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.452+0000 7f79723da640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f796c140180 0x7f796c140600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:55.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.452+0000 7f79723da640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f796c0b5630 con 0x7f79580052e0 2024-05-09T05:00:55.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.452+0000 7f79723da640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f796c0b54b0 con 0x7f796c1621d0 2024-05-09T05:00:55.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.452+0000 7f79723da640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f796c0b57b0 con 0x7f796c140180 2024-05-09T05:00:55.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.452+0000 7f796bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79580052e0 0x7f796c13c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:55.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.452+0000 7f796bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79580052e0 0x7f796c13c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51138/0 (socket says 172.21.15.60:51138) 2024-05-09T05:00:55.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.452+0000 7f796bfff640 1 -- 172.21.15.60:0/2434693358 learned_addr learned my addr 172.21.15.60:0/2434693358 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:55.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.452+0000 7f7970977640 1 --2- 172.21.15.60:0/2434693358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f796c140180 0x7f796c140600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:55.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.452+0000 7f7963fff640 1 --2- 172.21.15.60:0/2434693358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f796c1621d0 0x7f796c13b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:55.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.453+0000 7f796bfff640 1 -- 172.21.15.60:0/2434693358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f796c140180 msgr2=0x7f796c140600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:55.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.453+0000 7f796bfff640 1 --2- 172.21.15.60:0/2434693358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f796c140180 0x7f796c140600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:55.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.453+0000 7f796bfff640 1 -- 172.21.15.60:0/2434693358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f796c1621d0 msgr2=0x7f796c13b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:55.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.453+0000 7f796bfff640 1 --2- 172.21.15.60:0/2434693358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f796c1621d0 0x7f796c13b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:55.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.453+0000 7f796bfff640 1 -- 172.21.15.60:0/2434693358 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f796c13c660 con 0x7f79580052e0 2024-05-09T05:00:55.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.453+0000 7f796bfff640 1 --2- 172.21.15.60:0/2434693358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79580052e0 0x7f796c13c080 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f795c007310 tx=0x7f795c0077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:55.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.453+0000 7f79697fa640 1 -- 172.21.15.60:0/2434693358 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c01a070 con 0x7f79580052e0 2024-05-09T05:00:55.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.453+0000 7f79697fa640 1 -- 172.21.15.60:0/2434693358 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f795c00f940 con 0x7f79580052e0 2024-05-09T05:00:55.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.453+0000 7f79697fa640 1 -- 172.21.15.60:0/2434693358 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c00fd30 con 0x7f79580052e0 2024-05-09T05:00:55.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.453+0000 7f79723da640 1 -- 172.21.15.60:0/2434693358 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f796c13c940 con 0x7f79580052e0 2024-05-09T05:00:55.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.453+0000 7f79723da640 1 -- 172.21.15.60:0/2434693358 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f796c13cf00 con 0x7f79580052e0 2024-05-09T05:00:55.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.454+0000 7f79723da640 1 -- 172.21.15.60:0/2434693358 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f796c051040 con 0x7f79580052e0 2024-05-09T05:00:55.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.454+0000 7f79697fa640 1 -- 172.21.15.60:0/2434693358 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f795c01d070 con 0x7f79580052e0 2024-05-09T05:00:55.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.455+0000 7f79697fa640 1 --2- 172.21.15.60:0/2434693358 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f794c076300 0x7f794c0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:55.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.455+0000 7f79697fa640 1 -- 172.21.15.60:0/2434693358 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6618+0+0 (secure 0 0 0) 0x7f795c0972a0 con 0x7f79580052e0 2024-05-09T05:00:55.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.455+0000 7f7963fff640 1 --2- 172.21.15.60:0/2434693358 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f794c076300 0x7f794c0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:55.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.456+0000 7f7963fff640 1 --2- 172.21.15.60:0/2434693358 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f794c076300 0x7f794c0787c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f796c0badf0 tx=0x7f7954008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:55.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.459+0000 7f79697fa640 1 -- 172.21.15.60:0/2434693358 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f795c060870 con 0x7f79580052e0 2024-05-09T05:00:55.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:55.650+0000 7f79723da640 1 -- 172.21.15.60:0/2434693358 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f796c15f210 con 0x7f79580052e0 2024-05-09T05:00:56.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.280+0000 7f79697fa640 1 -- 172.21.15.60:0/2434693358 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v26) ==== 181+0+0 (secure 0 0 0) 0x7f795c0970c0 con 0x7f79580052e0 2024-05-09T05:00:56.281 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2024-05-09T05:00:56.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.283+0000 7f79723da640 1 -- 172.21.15.60:0/2434693358 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f794c076300 msgr2=0x7f794c0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:56.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.283+0000 7f79723da640 1 --2- 172.21.15.60:0/2434693358 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f794c076300 0x7f794c0787c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f796c0badf0 tx=0x7f7954008040 comp rx=0 tx=0).stop 2024-05-09T05:00:56.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.283+0000 7f79723da640 1 -- 172.21.15.60:0/2434693358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79580052e0 msgr2=0x7f796c13c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:56.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.283+0000 7f79723da640 1 --2- 172.21.15.60:0/2434693358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79580052e0 0x7f796c13c080 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f795c007310 tx=0x7f795c0077d0 comp rx=0 tx=0).stop 2024-05-09T05:00:56.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.283+0000 7f79723da640 1 -- 172.21.15.60:0/2434693358 shutdown_connections 2024-05-09T05:00:56.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.283+0000 7f79723da640 1 --2- 172.21.15.60:0/2434693358 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f794c076300 0x7f794c0787c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:56.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.283+0000 7f79723da640 1 --2- 172.21.15.60:0/2434693358 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f796c140180 0x7f796c140600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:56.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.283+0000 7f79723da640 1 --2- 172.21.15.60:0/2434693358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f796c1621d0 0x7f796c13b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:56.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.283+0000 7f79723da640 1 --2- 172.21.15.60:0/2434693358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79580052e0 0x7f796c13c080 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:56.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.283+0000 7f79723da640 1 -- 172.21.15.60:0/2434693358 >> 172.21.15.60:0/2434693358 conn(0x7f796c06f640 msgr2=0x7f796c0d45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:56.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.283+0000 7f79723da640 1 -- 172.21.15.60:0/2434693358 shutdown_connections 2024-05-09T05:00:56.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.283+0000 7f79723da640 1 -- 172.21.15.60:0/2434693358 wait complete. 2024-05-09T05:00:56.299 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-09T05:00:56.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.413+0000 7f7e7c503640 1 Processor -- start 2024-05-09T05:00:56.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.413+0000 7f7e7c503640 1 -- start start 2024-05-09T05:00:56.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e7c503640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f7e7415f0f0 0x7f7e740b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:56.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e7c503640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e740b54b0 0x7f7e740b8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:56.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e7c503640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e740b5990 con 0x7f7e7415f4d0 2024-05-09T05:00:56.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e7c503640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f7e740b65a0 con 0x7f7e7415f0f0 2024-05-09T05:00:56.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e7c503640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7e740b5c70 con 0x7f7e740b54b0 2024-05-09T05:00:56.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e7aaa0640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7e7415f4d0 0x7f7e740ba1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56498/0 (socket says 172.21.15.60:56498) 2024-05-09T05:00:56.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e7aaa0640 1 -- 172.21.15.60:0/87593742 learned_addr learned my addr 172.21.15.60:0/87593742 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:56.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e79a9e640 1 --2- 172.21.15.60:0/87593742 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e740b54b0 0x7f7e740b8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:56.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e7a29f640 1 --2- 172.21.15.60:0/87593742 >> v2:172.21.15.114:3300/0 conn(0x7f7e7415f0f0 0x7f7e740b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:56.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e7929d640 1 -- 172.21.15.60:0/87593742 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4285286734 0 0) 0x7f7e740b5990 con 0x7f7e7415f4d0 2024-05-09T05:00:56.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e7929d640 1 -- 172.21.15.60:0/87593742 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e60003520 con 0x7f7e7415f4d0 2024-05-09T05:00:56.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e7929d640 1 -- 172.21.15.60:0/87593742 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1403593541 0 0) 0x7f7e60003520 con 0x7f7e7415f4d0 2024-05-09T05:00:56.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e7929d640 1 -- 172.21.15.60:0/87593742 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e740b54b0 msgr2=0x7f7e740b8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:56.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e7929d640 1 --2- 172.21.15.60:0/87593742 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e740b54b0 0x7f7e740b8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:56.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e7929d640 1 -- 172.21.15.60:0/87593742 >> v2:172.21.15.114:3300/0 conn(0x7f7e7415f0f0 msgr2=0x7f7e740b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:56.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e7929d640 1 --2- 172.21.15.60:0/87593742 >> v2:172.21.15.114:3300/0 conn(0x7f7e7415f0f0 0x7f7e740b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:56.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e7929d640 1 -- 172.21.15.60:0/87593742 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e740b5f50 con 0x7f7e7415f4d0 2024-05-09T05:00:56.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e7929d640 1 -- 172.21.15.60:0/87593742 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f7e6c001a70 con 0x7f7e7415f4d0 2024-05-09T05:00:56.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e7929d640 1 -- 172.21.15.60:0/87593742 >> v1:172.21.15.60:6789/0 conn(0x7f7e7415f4d0 legacy=0x7f7e740ba1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.414+0000 7f7e7929d640 1 --2- 172.21.15.60:0/87593742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e60003a00 0x7f7e60003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.415+0000 7f7e7929d640 1 --2- 172.21.15.60:0/87593742 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e740b54b0 0x7f7e60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.415+0000 7f7e7929d640 1 --2- 172.21.15.60:0/87593742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e60004d40 0x7f7e600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.415+0000 7f7e7929d640 1 -- 172.21.15.60:0/87593742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7e740b65a0 con 0x7f7e60004d40 2024-05-09T05:00:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.415+0000 7f7e7929d640 1 -- 172.21.15.60:0/87593742 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7e740b5c70 con 0x7f7e60003a00 2024-05-09T05:00:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.415+0000 7f7e7929d640 1 -- 172.21.15.60:0/87593742 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7e6000b9f0 con 0x7f7e740b54b0 2024-05-09T05:00:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.415+0000 7f7e7a29f640 1 --2- 172.21.15.60:0/87593742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e60003a00 0x7f7e60003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.415+0000 7f7e7aaa0640 1 --2- 172.21.15.60:0/87593742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e60004d40 0x7f7e600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.415+0000 7f7e79a9e640 1 --2- 172.21.15.60:0/87593742 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e740b54b0 0x7f7e60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.415+0000 7f7e7aaa0640 1 -- 172.21.15.60:0/87593742 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e740b54b0 msgr2=0x7f7e60004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.415+0000 7f7e7aaa0640 1 --2- 172.21.15.60:0/87593742 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e740b54b0 0x7f7e60004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.415+0000 7f7e7aaa0640 1 -- 172.21.15.60:0/87593742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e60003a00 msgr2=0x7f7e60003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.415+0000 7f7e7aaa0640 1 --2- 172.21.15.60:0/87593742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e60003a00 0x7f7e60003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.415+0000 7f7e7aaa0640 1 -- 172.21.15.60:0/87593742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e60003520 con 0x7f7e60004d40 2024-05-09T05:00:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.415+0000 7f7e7a29f640 1 --2- 172.21.15.60:0/87593742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e60003a00 0x7f7e60003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:00:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.416+0000 7f7e7aaa0640 1 --2- 172.21.15.60:0/87593742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e60004d40 0x7f7e600051c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00a850 tx=0x7f7e60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53614500606e8fc5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.416+0000 7f7e7929d640 1 -- 172.21.15.60:0/87593742 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c00a4a0 con 0x7f7e60004d40 2024-05-09T05:00:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.416+0000 7f7e7929d640 1 -- 172.21.15.60:0/87593742 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e6c02f070 con 0x7f7e60004d40 2024-05-09T05:00:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.416+0000 7f7e7929d640 1 -- 172.21.15.60:0/87593742 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c02bc30 con 0x7f7e60004d40 2024-05-09T05:00:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.416+0000 7f7e7c503640 1 -- 172.21.15.60:0/87593742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e60004d40 msgr2=0x7f7e600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.416+0000 7f7e7c503640 1 --2- 172.21.15.60:0/87593742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e60004d40 0x7f7e600051c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00a850 tx=0x7f7e60005830 comp rx=0 tx=0).stop 2024-05-09T05:00:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.416+0000 7f7e7aaa0640 1 -- 172.21.15.60:0/87593742 reap_dead start 2024-05-09T05:00:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.417+0000 7f7e7c503640 1 -- 172.21.15.60:0/87593742 shutdown_connections 2024-05-09T05:00:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.417+0000 7f7e7c503640 1 -- 172.21.15.60:0/87593742 >> 172.21.15.60:0/87593742 conn(0x7f7e7406ede0 msgr2=0x7f7e74073820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.417+0000 7f7e7c503640 1 -- 172.21.15.60:0/87593742 shutdown_connections 2024-05-09T05:00:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.417+0000 7f7e7c503640 1 -- 172.21.15.60:0/87593742 wait complete. 2024-05-09T05:00:56.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.418+0000 7f7e7c503640 1 Processor -- start 2024-05-09T05:00:56.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.418+0000 7f7e7c503640 1 -- start start 2024-05-09T05:00:56.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e7c503640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e60004d40 0x7f7e74152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:56.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e7c503640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e741f8f50 0x7f7e74154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:56.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e7c503640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e741f93d0 0x7f7e7415de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:56.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e7c503640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7e740baa30 con 0x7f7e741f93d0 2024-05-09T05:00:56.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e7c503640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7e740ba8b0 con 0x7f7e741f8f50 2024-05-09T05:00:56.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e7c503640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7e740b8f60 con 0x7f7e60004d40 2024-05-09T05:00:56.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e7aaa0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e741f93d0 0x7f7e7415de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:56.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e7aaa0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e741f93d0 0x7f7e7415de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51158/0 (socket says 172.21.15.60:51158) 2024-05-09T05:00:56.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e7aaa0640 1 -- 172.21.15.60:0/4059975240 learned_addr learned my addr 172.21.15.60:0/4059975240 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:56.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e7a29f640 1 --2- 172.21.15.60:0/4059975240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e60004d40 0x7f7e74152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:56.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e79a9e640 1 --2- 172.21.15.60:0/4059975240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e741f8f50 0x7f7e74154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:56.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e7aaa0640 1 -- 172.21.15.60:0/4059975240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e60004d40 msgr2=0x7f7e74152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:56.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e7aaa0640 1 --2- 172.21.15.60:0/4059975240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e60004d40 0x7f7e74152be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:56.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e7aaa0640 1 -- 172.21.15.60:0/4059975240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e741f8f50 msgr2=0x7f7e74154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:56.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e7aaa0640 1 --2- 172.21.15.60:0/4059975240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e741f8f50 0x7f7e74154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:56.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e7aaa0640 1 -- 172.21.15.60:0/4059975240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e741f9930 con 0x7f7e741f93d0 2024-05-09T05:00:56.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e7aaa0640 1 --2- 172.21.15.60:0/4059975240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e741f93d0 0x7f7e7415de40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00a970 tx=0x7f7e740b6120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:56.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e6affd640 1 -- 172.21.15.60:0/4059975240 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c044070 con 0x7f7e741f93d0 2024-05-09T05:00:56.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e6affd640 1 -- 172.21.15.60:0/4059975240 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e6c00a480 con 0x7f7e741f93d0 2024-05-09T05:00:56.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.419+0000 7f7e6affd640 1 -- 172.21.15.60:0/4059975240 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e70009690 con 0x7f7e741f93d0 2024-05-09T05:00:56.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.420+0000 7f7e7c503640 1 -- 172.21.15.60:0/4059975240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e741f9c10 con 0x7f7e741f93d0 2024-05-09T05:00:56.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.420+0000 7f7e7c503640 1 -- 172.21.15.60:0/4059975240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e7415d640 con 0x7f7e741f93d0 2024-05-09T05:00:56.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.420+0000 7f7e7c503640 1 -- 172.21.15.60:0/4059975240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e74051040 con 0x7f7e741f93d0 2024-05-09T05:00:56.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.421+0000 7f7e6affd640 1 -- 172.21.15.60:0/4059975240 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7e6c047070 con 0x7f7e741f93d0 2024-05-09T05:00:56.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.422+0000 7f7e6affd640 1 --2- 172.21.15.60:0/4059975240 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7e580762d0 0x7f7e58078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:56.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.422+0000 7f7e6affd640 1 -- 172.21.15.60:0/4059975240 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6180+0+0 (secure 0 0 0) 0x7f7e6c0b7580 con 0x7f7e741f93d0 2024-05-09T05:00:56.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.422+0000 7f7e7a29f640 1 --2- 172.21.15.60:0/4059975240 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7e580762d0 0x7f7e58078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:56.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.422+0000 7f7e7a29f640 1 --2- 172.21.15.60:0/4059975240 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7e580762d0 0x7f7e58078790 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7e740d1fa0 tx=0x7f7e70009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:56.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.426+0000 7f7e6affd640 1 -- 172.21.15.60:0/4059975240 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7e6c0806d0 con 0x7f7e741f93d0 2024-05-09T05:00:56.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:56.619+0000 7f7e7c503640 1 -- 172.21.15.60:0/4059975240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7e741f9de0 con 0x7f7e741f93d0 2024-05-09T05:00:57.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.285+0000 7f7e6affd640 1 -- 172.21.15.60:0/4059975240 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v27) ==== 169+0+0 (secure 0 0 0) 0x7f7e6c0849d0 con 0x7f7e741f93d0 2024-05-09T05:00:57.286 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2024-05-09T05:00:57.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.290+0000 7f7e7c503640 1 -- 172.21.15.60:0/4059975240 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7e580762d0 msgr2=0x7f7e58078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:57.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.290+0000 7f7e7c503640 1 --2- 172.21.15.60:0/4059975240 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7e580762d0 0x7f7e58078790 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7e740d1fa0 tx=0x7f7e70009190 comp rx=0 tx=0).stop 2024-05-09T05:00:57.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.290+0000 7f7e7c503640 1 -- 172.21.15.60:0/4059975240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e741f93d0 msgr2=0x7f7e7415de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:57.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.290+0000 7f7e7c503640 1 --2- 172.21.15.60:0/4059975240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e741f93d0 0x7f7e7415de40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00a970 tx=0x7f7e740b6120 comp rx=0 tx=0).stop 2024-05-09T05:00:57.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.290+0000 7f7e7c503640 1 -- 172.21.15.60:0/4059975240 shutdown_connections 2024-05-09T05:00:57.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.290+0000 7f7e7c503640 1 --2- 172.21.15.60:0/4059975240 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7e580762d0 0x7f7e58078790 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:57.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.290+0000 7f7e7c503640 1 --2- 172.21.15.60:0/4059975240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e741f93d0 0x7f7e7415de40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:57.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.290+0000 7f7e7c503640 1 --2- 172.21.15.60:0/4059975240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e741f8f50 0x7f7e74154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:57.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.290+0000 7f7e7c503640 1 --2- 172.21.15.60:0/4059975240 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7e60004d40 0x7f7e74152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:57.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.290+0000 7f7e7c503640 1 -- 172.21.15.60:0/4059975240 >> 172.21.15.60:0/4059975240 conn(0x7f7e7406ede0 msgr2=0x7f7e740d4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:57.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.290+0000 7f7e7c503640 1 -- 172.21.15.60:0/4059975240 shutdown_connections 2024-05-09T05:00:57.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.290+0000 7f7e7c503640 1 -- 172.21.15.60:0/4059975240 wait complete. 2024-05-09T05:00:57.307 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-05-09T05:00:57.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.421+0000 7fc31bb19640 1 Processor -- start 2024-05-09T05:00:57.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.421+0000 7fc31bb19640 1 -- start start 2024-05-09T05:00:57.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.422+0000 7fc31bb19640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc314072ce0 0x7fc31415ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:57.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.422+0000 7fc31bb19640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc3140b06f0 0x7fc31406e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:57.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.422+0000 7fc31bb19640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc314070940 con 0x7fc314071810 2024-05-09T05:00:57.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.422+0000 7fc31bb19640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fc314072620 con 0x7fc314072ce0 2024-05-09T05:00:57.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.422+0000 7fc31bb19640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fc3140b0d60 con 0x7fc3140b06f0 2024-05-09T05:00:57.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.422+0000 7fc3198b5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc314072ce0 0x7fc31415ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:57.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.422+0000 7fc31a0b6640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc314071810 0x7fc3140733c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56502/0 (socket says 172.21.15.60:56502) 2024-05-09T05:00:57.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.422+0000 7fc31a0b6640 1 -- 172.21.15.60:0/3537189406 learned_addr learned my addr 172.21.15.60:0/3537189406 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:57.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.423+0000 7fc3190b4640 1 --2- 172.21.15.60:0/3537189406 >> v2:172.21.15.114:3300/0 conn(0x7fc3140b06f0 0x7fc31406e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:57.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.423+0000 7fc3198b5640 1 -- 172.21.15.60:0/3537189406 >> v2:172.21.15.114:3300/0 conn(0x7fc3140b06f0 msgr2=0x7fc31406e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:57.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.423+0000 7fc3198b5640 1 --2- 172.21.15.60:0/3537189406 >> v2:172.21.15.114:3300/0 conn(0x7fc3140b06f0 0x7fc31406e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:57.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.423+0000 7fc3198b5640 1 -- 172.21.15.60:0/3537189406 >> v1:172.21.15.60:6789/0 conn(0x7fc314071810 legacy=0x7fc3140733c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:57.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.423+0000 7fc3188b3640 1 -- 172.21.15.60:0/3537189406 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2968970488 0 0) 0x7fc314070940 con 0x7fc314071810 2024-05-09T05:00:57.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.423+0000 7fc3198b5640 1 -- 172.21.15.60:0/3537189406 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3140d0470 con 0x7fc314072ce0 2024-05-09T05:00:57.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.424+0000 7fc3198b5640 1 --2- 172.21.15.60:0/3537189406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc314072ce0 0x7fc31415ed00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc3140b5350 tx=0x7fc30c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df78b75cdebd0d99 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:57.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.424+0000 7fc3188b3640 1 -- 172.21.15.60:0/3537189406 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c037070 con 0x7fc314072ce0 2024-05-09T05:00:57.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.424+0000 7fc3188b3640 1 -- 172.21.15.60:0/3537189406 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc30c032440 con 0x7fc314072ce0 2024-05-09T05:00:57.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.424+0000 7fc3188b3640 1 -- 172.21.15.60:0/3537189406 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c032810 con 0x7fc314072ce0 2024-05-09T05:00:57.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.424+0000 7fc31bb19640 1 -- 172.21.15.60:0/3537189406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc314072ce0 msgr2=0x7fc31415ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:57.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.424+0000 7fc31bb19640 1 --2- 172.21.15.60:0/3537189406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc314072ce0 0x7fc31415ed00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc3140b5350 tx=0x7fc30c02a1a0 comp rx=0 tx=0).stop 2024-05-09T05:00:57.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.424+0000 7fc31bb19640 1 -- 172.21.15.60:0/3537189406 shutdown_connections 2024-05-09T05:00:57.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.424+0000 7fc31bb19640 1 --2- 172.21.15.60:0/3537189406 >> v2:172.21.15.114:3300/0 conn(0x7fc3140b06f0 0x7fc31406e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:57.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.424+0000 7fc31bb19640 1 --2- 172.21.15.60:0/3537189406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc314072ce0 0x7fc31415ed00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:57.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.424+0000 7fc31bb19640 1 -- 172.21.15.60:0/3537189406 >> 172.21.15.60:0/3537189406 conn(0x7fc3140b0f60 msgr2=0x7fc3140b1370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:57.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.424+0000 7fc31bb19640 1 -- 172.21.15.60:0/3537189406 shutdown_connections 2024-05-09T05:00:57.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.424+0000 7fc31bb19640 1 -- 172.21.15.60:0/3537189406 wait complete. 2024-05-09T05:00:57.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.426+0000 7fc31bb19640 1 Processor -- start 2024-05-09T05:00:57.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.426+0000 7fc31bb19640 1 -- start start 2024-05-09T05:00:57.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.426+0000 7fc31bb19640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc314071810 0x7fc3140aa720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:57.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.426+0000 7fc31bb19640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc314072ce0 0x7fc3141f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:57.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.426+0000 7fc31bb19640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3140b06f0 0x7fc31415ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:57.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.426+0000 7fc31bb19640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc314070140 con 0x7fc314071810 2024-05-09T05:00:57.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.426+0000 7fc31bb19640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc31406fe60 con 0x7fc314072ce0 2024-05-09T05:00:57.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.426+0000 7fc31bb19640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc314070420 con 0x7fc3140b06f0 2024-05-09T05:00:57.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.426+0000 7fc3198b5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc314071810 0x7fc3140aa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:57.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.427+0000 7fc3198b5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc314071810 0x7fc3140aa720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51182/0 (socket says 172.21.15.60:51182) 2024-05-09T05:00:57.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.427+0000 7fc3198b5640 1 -- 172.21.15.60:0/1494225669 learned_addr learned my addr 172.21.15.60:0/1494225669 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:57.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.427+0000 7fc3190b4640 1 --2- 172.21.15.60:0/1494225669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc314072ce0 0x7fc3141f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:57.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.427+0000 7fc31a0b6640 1 --2- 172.21.15.60:0/1494225669 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3140b06f0 0x7fc31415ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:57.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.427+0000 7fc3198b5640 1 -- 172.21.15.60:0/1494225669 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3140b06f0 msgr2=0x7fc31415ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:57.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.427+0000 7fc3198b5640 1 --2- 172.21.15.60:0/1494225669 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3140b06f0 0x7fc31415ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:57.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.427+0000 7fc3198b5640 1 -- 172.21.15.60:0/1494225669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc314072ce0 msgr2=0x7fc3141f6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:57.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.427+0000 7fc3198b5640 1 --2- 172.21.15.60:0/1494225669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc314072ce0 0x7fc3141f6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:57.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.427+0000 7fc3198b5640 1 -- 172.21.15.60:0/1494225669 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3141577d0 con 0x7fc314071810 2024-05-09T05:00:57.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.427+0000 7fc31a0b6640 1 --2- 172.21.15.60:0/1494225669 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3140b06f0 0x7fc31415ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:57.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.427+0000 7fc3198b5640 1 --2- 172.21.15.60:0/1494225669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc314071810 0x7fc3140aa720 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc30c000c00 tx=0x7fc30c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:57.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.427+0000 7fc3127fc640 1 -- 172.21.15.60:0/1494225669 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c037040 con 0x7fc314071810 2024-05-09T05:00:57.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.427+0000 7fc3127fc640 1 -- 172.21.15.60:0/1494225669 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc30c042070 con 0x7fc314071810 2024-05-09T05:00:57.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.427+0000 7fc31bb19640 1 -- 172.21.15.60:0/1494225669 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc314157ab0 con 0x7fc314071810 2024-05-09T05:00:57.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.427+0000 7fc3127fc640 1 -- 172.21.15.60:0/1494225669 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c030bd0 con 0x7fc314071810 2024-05-09T05:00:57.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.427+0000 7fc31bb19640 1 -- 172.21.15.60:0/1494225669 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc314158070 con 0x7fc314071810 2024-05-09T05:00:57.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.428+0000 7fc31bb19640 1 -- 172.21.15.60:0/1494225669 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc314051040 con 0x7fc314071810 2024-05-09T05:00:57.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.429+0000 7fc3127fc640 1 -- 172.21.15.60:0/1494225669 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc30c047460 con 0x7fc314071810 2024-05-09T05:00:57.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.429+0000 7fc3127fc640 1 --2- 172.21.15.60:0/1494225669 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc3000762d0 0x7fc300078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:57.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.430+0000 7fc3127fc640 1 -- 172.21.15.60:0/1494225669 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5766+0+0 (secure 0 0 0) 0x7fc30c0b82a0 con 0x7fc314071810 2024-05-09T05:00:57.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.430+0000 7fc3190b4640 1 --2- 172.21.15.60:0/1494225669 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc3000762d0 0x7fc300078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:57.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.430+0000 7fc3190b4640 1 --2- 172.21.15.60:0/1494225669 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc3000762d0 0x7fc300078790 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc314155820 tx=0x7fc308008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:57.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.434+0000 7fc3127fc640 1 -- 172.21.15.60:0/1494225669 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc30c032e20 con 0x7fc314071810 2024-05-09T05:00:57.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:57.628+0000 7fc31bb19640 1 -- 172.21.15.60:0/1494225669 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc3140cd620 con 0x7fc314071810 2024-05-09T05:00:58.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.287+0000 7fc3127fc640 1 -- 172.21.15.60:0/1494225669 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v28) ==== 178+0+0 (secure 0 0 0) 0x7fc30c081b20 con 0x7fc314071810 2024-05-09T05:00:58.287 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data_ec' removed 2024-05-09T05:00:58.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.291+0000 7fc31bb19640 1 -- 172.21.15.60:0/1494225669 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc3000762d0 msgr2=0x7fc300078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:58.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.291+0000 7fc31bb19640 1 --2- 172.21.15.60:0/1494225669 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc3000762d0 0x7fc300078790 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc314155820 tx=0x7fc308008040 comp rx=0 tx=0).stop 2024-05-09T05:00:58.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.291+0000 7fc31bb19640 1 -- 172.21.15.60:0/1494225669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc314071810 msgr2=0x7fc3140aa720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:58.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.291+0000 7fc31bb19640 1 --2- 172.21.15.60:0/1494225669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc314071810 0x7fc3140aa720 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc30c000c00 tx=0x7fc30c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:00:58.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.291+0000 7fc31bb19640 1 -- 172.21.15.60:0/1494225669 shutdown_connections 2024-05-09T05:00:58.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.291+0000 7fc31bb19640 1 --2- 172.21.15.60:0/1494225669 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc3000762d0 0x7fc300078790 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:58.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.291+0000 7fc31bb19640 1 --2- 172.21.15.60:0/1494225669 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc3140b06f0 0x7fc31415ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:58.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.291+0000 7fc31bb19640 1 --2- 172.21.15.60:0/1494225669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc314072ce0 0x7fc3141f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:58.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.291+0000 7fc31bb19640 1 --2- 172.21.15.60:0/1494225669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc314071810 0x7fc3140aa720 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:58.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.291+0000 7fc31bb19640 1 -- 172.21.15.60:0/1494225669 >> 172.21.15.60:0/1494225669 conn(0x7fc3140b0f60 msgr2=0x7fc314155fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:58.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.291+0000 7fc31bb19640 1 -- 172.21.15.60:0/1494225669 shutdown_connections 2024-05-09T05:00:58.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.291+0000 7fc31bb19640 1 -- 172.21.15.60:0/1494225669 wait complete. 2024-05-09T05:00:58.308 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-09T05:00:58.309 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-09T05:00:58.309 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-09T05:00:58.309 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:00:58.309 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-09T05:00:58.309 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-09T05:00:58.310 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-09T05:00:58.310 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:00:58.310 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-09T05:00:58.310 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-09T05:00:58.310 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-09T05:00:58.310 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:00:58.310 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-09T05:00:58.311 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-09T05:00:58.311 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-09T05:00:58.311 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:00:58.320 INFO:tasks.ceph.mds.a:Stopped 2024-05-09T05:00:58.320 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-09T05:00:58.321 INFO:tasks.ceph.mds.c:Stopped 2024-05-09T05:00:58.321 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-09T05:00:58.322 INFO:tasks.ceph.mds.b:Stopped 2024-05-09T05:00:58.322 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-09T05:00:58.323 INFO:tasks.ceph.mds.d:Stopped 2024-05-09T05:00:58.323 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-09T05:00:58.363 INFO:tasks.ceph.mds.a:Started 2024-05-09T05:00:58.364 DEBUG:teuthology.parallel:result is None 2024-05-09T05:00:58.364 INFO:tasks.ceph.mds.c:Started 2024-05-09T05:00:58.364 DEBUG:teuthology.parallel:result is None 2024-05-09T05:00:58.365 INFO:tasks.ceph.mds.b:Started 2024-05-09T05:00:58.366 INFO:tasks.ceph.mds.d:Started 2024-05-09T05:00:58.366 DEBUG:teuthology.parallel:result is None 2024-05-09T05:00:58.366 DEBUG:teuthology.parallel:result is None 2024-05-09T05:00:58.366 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-09T05:00:58.446 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2024-05-09T05:00:58.446 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2024-05-09T05:00:58.451 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2024-05-09T05:00:58.451 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2024-05-09T05:00:58.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.489+0000 7fc7a0571640 1 Processor -- start 2024-05-09T05:00:58.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.489+0000 7fc7a0571640 1 -- start start 2024-05-09T05:00:58.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.490+0000 7fc7a0571640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc79806fe30 0x7fc798070290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:58.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.490+0000 7fc7a0571640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc798153360 con 0x7fc798152f80 2024-05-09T05:00:58.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.490+0000 7fc7a0571640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc79815c8b0 con 0x7fc7980d0480 2024-05-09T05:00:58.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.490+0000 7fc7a0571640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc79815cb90 con 0x7fc79806fe30 2024-05-09T05:00:58.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.490+0000 7fc79e30d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc798152f80 0x7fc7980ad800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56518/0 (socket says 172.21.15.60:56518) 2024-05-09T05:00:58.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.490+0000 7fc79e30d640 1 -- 172.21.15.60:0/879571835 learned_addr learned my addr 172.21.15.60:0/879571835 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:58.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.490+0000 7fc79db0c640 1 --2- 172.21.15.60:0/879571835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc79806fe30 0x7fc798070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:58.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.491+0000 7fc79d30b640 1 -- 172.21.15.60:0/879571835 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 745104405 0 0) 0x7fc798153360 con 0x7fc798152f80 2024-05-09T05:00:58.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.491+0000 7fc79d30b640 1 -- 172.21.15.60:0/879571835 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc784003520 con 0x7fc798152f80 2024-05-09T05:00:58.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.491+0000 7fc79d30b640 1 -- 172.21.15.60:0/879571835 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2241369697 0 0) 0x7fc79815c8b0 con 0x7fc7980d0480 2024-05-09T05:00:58.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.491+0000 7fc79d30b640 1 -- 172.21.15.60:0/879571835 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc798153360 con 0x7fc7980d0480 2024-05-09T05:00:58.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.491+0000 7fc79db0c640 1 -- 172.21.15.60:0/879571835 >> v1:172.21.15.114:6789/0 conn(0x7fc7980d0480 legacy=0x7fc7980b10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:58.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.491+0000 7fc79db0c640 1 -- 172.21.15.60:0/879571835 >> v1:172.21.15.60:6789/0 conn(0x7fc798152f80 legacy=0x7fc7980ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:58.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.491+0000 7fc79db0c640 1 -- 172.21.15.60:0/879571835 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc79815ce70 con 0x7fc79806fe30 2024-05-09T05:00:58.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.492+0000 7fc79db0c640 1 --2- 172.21.15.60:0/879571835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc79806fe30 0x7fc798070290 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc7880097d0 tx=0x7fc78802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f863237d87d54bb2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:58.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.492+0000 7fc79d30b640 1 -- 172.21.15.60:0/879571835 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc788037070 con 0x7fc79806fe30 2024-05-09T05:00:58.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.492+0000 7fc79d30b640 1 -- 172.21.15.60:0/879571835 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc788032440 con 0x7fc79806fe30 2024-05-09T05:00:58.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.492+0000 7fc79d30b640 1 -- 172.21.15.60:0/879571835 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc788032810 con 0x7fc79806fe30 2024-05-09T05:00:58.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.492+0000 7fc7a0571640 1 -- 172.21.15.60:0/879571835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc79806fe30 msgr2=0x7fc798070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:58.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.492+0000 7fc7a0571640 1 --2- 172.21.15.60:0/879571835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc79806fe30 0x7fc798070290 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc7880097d0 tx=0x7fc78802a180 comp rx=0 tx=0).stop 2024-05-09T05:00:58.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.492+0000 7fc7a0571640 1 -- 172.21.15.60:0/879571835 shutdown_connections 2024-05-09T05:00:58.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.492+0000 7fc7a0571640 1 --2- 172.21.15.60:0/879571835 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc79806fe30 0x7fc798070290 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:58.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.492+0000 7fc7a0571640 1 -- 172.21.15.60:0/879571835 >> 172.21.15.60:0/879571835 conn(0x7fc7980b4850 msgr2=0x7fc7980b4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:58.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.493+0000 7fc7a0571640 1 -- 172.21.15.60:0/879571835 shutdown_connections 2024-05-09T05:00:58.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.493+0000 7fc7a0571640 1 -- 172.21.15.60:0/879571835 wait complete. 2024-05-09T05:00:58.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.494+0000 7fc7a0571640 1 Processor -- start 2024-05-09T05:00:58.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.494+0000 7fc7a0571640 1 -- start start 2024-05-09T05:00:58.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.494+0000 7fc7a0571640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc79806fe30 0x7fc7980a9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:58.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.494+0000 7fc7a0571640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7980d0480 0x7fc798156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:58.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.494+0000 7fc7a0571640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc798152f80 0x7fc798141270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:58.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.494+0000 7fc7a0571640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc79815c670 con 0x7fc79806fe30 2024-05-09T05:00:58.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.494+0000 7fc7a0571640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc79815c4f0 con 0x7fc798152f80 2024-05-09T05:00:58.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.494+0000 7fc7a0571640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc7980b1800 con 0x7fc7980d0480 2024-05-09T05:00:58.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.495+0000 7fc79e30d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc79806fe30 0x7fc7980a9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:58.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.495+0000 7fc79e30d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc79806fe30 0x7fc7980a9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51200/0 (socket says 172.21.15.60:51200) 2024-05-09T05:00:58.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.495+0000 7fc79e30d640 1 -- 172.21.15.60:0/526605242 learned_addr learned my addr 172.21.15.60:0/526605242 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:58.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.495+0000 7fc79db0c640 1 --2- 172.21.15.60:0/526605242 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7980d0480 0x7fc798156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:58.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.495+0000 7fc79eb0e640 1 --2- 172.21.15.60:0/526605242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc798152f80 0x7fc798141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:58.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.495+0000 7fc79e30d640 1 -- 172.21.15.60:0/526605242 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7980d0480 msgr2=0x7fc798156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:58.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.495+0000 7fc79e30d640 1 --2- 172.21.15.60:0/526605242 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7980d0480 0x7fc798156b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:58.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.495+0000 7fc79e30d640 1 -- 172.21.15.60:0/526605242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc798152f80 msgr2=0x7fc798141270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:58.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.495+0000 7fc79e30d640 1 --2- 172.21.15.60:0/526605242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc798152f80 0x7fc798141270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:58.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.495+0000 7fc79e30d640 1 -- 172.21.15.60:0/526605242 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc798156040 con 0x7fc79806fe30 2024-05-09T05:00:58.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.495+0000 7fc79e30d640 1 --2- 172.21.15.60:0/526605242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc79806fe30 0x7fc7980a9ce0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc79400c410 tx=0x7fc79400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:58.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.495+0000 7fc78effd640 1 -- 172.21.15.60:0/526605242 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc794017070 con 0x7fc79806fe30 2024-05-09T05:00:58.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.495+0000 7fc78effd640 1 -- 172.21.15.60:0/526605242 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc794005da0 con 0x7fc79806fe30 2024-05-09T05:00:58.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.495+0000 7fc7a0571640 1 -- 172.21.15.60:0/526605242 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc798156590 con 0x7fc79806fe30 2024-05-09T05:00:58.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.495+0000 7fc78effd640 1 -- 172.21.15.60:0/526605242 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7940072f0 con 0x7fc79806fe30 2024-05-09T05:00:58.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.495+0000 7fc7a0571640 1 -- 172.21.15.60:0/526605242 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc79813f3e0 con 0x7fc79806fe30 2024-05-09T05:00:58.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.497+0000 7fc7a0571640 1 -- 172.21.15.60:0/526605242 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7980b03b0 con 0x7fc79806fe30 2024-05-09T05:00:58.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.497+0000 7fc78effd640 1 -- 172.21.15.60:0/526605242 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc79401a070 con 0x7fc79806fe30 2024-05-09T05:00:58.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.498+0000 7fc78effd640 1 --2- 172.21.15.60:0/526605242 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc77c0762d0 0x7fc77c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:58.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.498+0000 7fc78effd640 1 -- 172.21.15.60:0/526605242 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5335+0+0 (secure 0 0 0) 0x7fc794096870 con 0x7fc79806fe30 2024-05-09T05:00:58.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.498+0000 7fc79db0c640 1 --2- 172.21.15.60:0/526605242 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc77c0762d0 0x7fc77c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:58.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.498+0000 7fc79db0c640 1 --2- 172.21.15.60:0/526605242 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc77c0762d0 0x7fc77c078790 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc788034000 tx=0x7fc78802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:58.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.502+0000 7fc78effd640 1 -- 172.21.15.60:0/526605242 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc79405fc60 con 0x7fc79806fe30 2024-05-09T05:00:58.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:58.704+0000 7fc7a0571640 1 -- 172.21.15.60:0/526605242 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fc798051040 con 0x7fc79806fe30 2024-05-09T05:00:59.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.305+0000 7fc78effd640 1 -- 172.21.15.60:0/526605242 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v30) ==== 97+0+0 (secure 0 0 0) 0x7fc794063f60 con 0x7fc79806fe30 2024-05-09T05:00:59.306 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blocklist entries 2024-05-09T05:00:59.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.309+0000 7fc7a0571640 1 -- 172.21.15.60:0/526605242 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc77c0762d0 msgr2=0x7fc77c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:59.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.309+0000 7fc7a0571640 1 --2- 172.21.15.60:0/526605242 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc77c0762d0 0x7fc77c078790 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc788034000 tx=0x7fc78802e000 comp rx=0 tx=0).stop 2024-05-09T05:00:59.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.309+0000 7fc7a0571640 1 -- 172.21.15.60:0/526605242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc79806fe30 msgr2=0x7fc7980a9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:59.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.309+0000 7fc7a0571640 1 --2- 172.21.15.60:0/526605242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc79806fe30 0x7fc7980a9ce0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc79400c410 tx=0x7fc79400c8d0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.309+0000 7fc7a0571640 1 -- 172.21.15.60:0/526605242 shutdown_connections 2024-05-09T05:00:59.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.309+0000 7fc7a0571640 1 --2- 172.21.15.60:0/526605242 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc77c0762d0 0x7fc77c078790 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.309+0000 7fc7a0571640 1 --2- 172.21.15.60:0/526605242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc798152f80 0x7fc798141270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.309+0000 7fc7a0571640 1 --2- 172.21.15.60:0/526605242 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc7980d0480 0x7fc798156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.309+0000 7fc7a0571640 1 --2- 172.21.15.60:0/526605242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc79806fe30 0x7fc7980a9ce0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.309+0000 7fc7a0571640 1 -- 172.21.15.60:0/526605242 >> 172.21.15.60:0/526605242 conn(0x7fc7980b4850 msgr2=0x7fc79815d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:59.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.309+0000 7fc7a0571640 1 -- 172.21.15.60:0/526605242 shutdown_connections 2024-05-09T05:00:59.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.309+0000 7fc7a0571640 1 -- 172.21.15.60:0/526605242 wait complete. 2024-05-09T05:00:59.325 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-09T05:00:59.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.435+0000 7f16fc114640 1 Processor -- start 2024-05-09T05:00:59.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.435+0000 7f16fc114640 1 -- start start 2024-05-09T05:00:59.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.436+0000 7f16fc114640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f415f8e0 0x7f16f406fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:59.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.436+0000 7f16fc114640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16f415fd40 0x7f16f4070360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:59.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.436+0000 7f16fc114640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16f409a210 con 0x7f16f415f500 2024-05-09T05:00:59.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.436+0000 7f16fc114640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f16f409a4e0 con 0x7f16f415fd40 2024-05-09T05:00:59.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.436+0000 7f16fc114640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f16f4072ad0 con 0x7f16f415f8e0 2024-05-09T05:00:59.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.436+0000 7f16fa6b1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16f415fd40 0x7f16f4070360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:59.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.436+0000 7f16fa6b1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16f415fd40 0x7f16f4070360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56036/0 (socket says 172.21.15.60:56036) 2024-05-09T05:00:59.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.436+0000 7f16fa6b1640 1 -- 172.21.15.60:0/1335636589 learned_addr learned my addr 172.21.15.60:0/1335636589 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:59.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.436+0000 7f16f96af640 1 --2- 172.21.15.60:0/1335636589 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f415f8e0 0x7f16f406fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:59.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.436+0000 7f16fa6b1640 1 -- 172.21.15.60:0/1335636589 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f415f8e0 msgr2=0x7f16f406fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:59.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.436+0000 7f16fa6b1640 1 --2- 172.21.15.60:0/1335636589 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f415f8e0 0x7f16f406fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.436+0000 7f16fa6b1640 1 -- 172.21.15.60:0/1335636589 >> v1:172.21.15.114:6789/0 conn(0x7f16f415f500 legacy=0x7f16f4099b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:59.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.436+0000 7f16fa6b1640 1 -- 172.21.15.60:0/1335636589 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f4072db0 con 0x7f16f415fd40 2024-05-09T05:00:59.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.436+0000 7f16f96af640 1 --2- 172.21.15.60:0/1335636589 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f415f8e0 0x7f16f406fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:59.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.437+0000 7f16fa6b1640 1 --2- 172.21.15.60:0/1335636589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16f415fd40 0x7f16f4070360 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009e50 tx=0x7f16ec02a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da9fd2d1fb17d772 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:59.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.437+0000 7f16f8eae640 1 -- 172.21.15.60:0/1335636589 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec036070 con 0x7f16f415fd40 2024-05-09T05:00:59.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.437+0000 7f16f8eae640 1 -- 172.21.15.60:0/1335636589 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16ec031600 con 0x7f16f415fd40 2024-05-09T05:00:59.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.437+0000 7f16f8eae640 1 -- 172.21.15.60:0/1335636589 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec0319d0 con 0x7f16f415fd40 2024-05-09T05:00:59.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.438+0000 7f16fc114640 1 -- 172.21.15.60:0/1335636589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16f415fd40 msgr2=0x7f16f4070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:59.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.438+0000 7f16fc114640 1 --2- 172.21.15.60:0/1335636589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16f415fd40 0x7f16f4070360 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009e50 tx=0x7f16ec02a5f0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.438+0000 7f16fc114640 1 -- 172.21.15.60:0/1335636589 shutdown_connections 2024-05-09T05:00:59.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.438+0000 7f16fc114640 1 --2- 172.21.15.60:0/1335636589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16f415fd40 0x7f16f4070360 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.438+0000 7f16fc114640 1 --2- 172.21.15.60:0/1335636589 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f415f8e0 0x7f16f406fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.438+0000 7f16fc114640 1 -- 172.21.15.60:0/1335636589 >> 172.21.15.60:0/1335636589 conn(0x7f16f40b4ec0 msgr2=0x7f16f406e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:59.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.438+0000 7f16fc114640 1 -- 172.21.15.60:0/1335636589 shutdown_connections 2024-05-09T05:00:59.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.438+0000 7f16fc114640 1 -- 172.21.15.60:0/1335636589 wait complete. 2024-05-09T05:00:59.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.439+0000 7f16fc114640 1 Processor -- start 2024-05-09T05:00:59.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.439+0000 7f16fc114640 1 -- start start 2024-05-09T05:00:59.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16fc114640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16f415f500 0x7f16f40b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:59.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16fc114640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f415f8e0 0x7f16f40ad470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:59.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16fc114640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16f415fd40 0x7f16f40d2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:59.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16fc114640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f16f409c140 con 0x7f16f415fd40 2024-05-09T05:00:59.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16fc114640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f16f409bfc0 con 0x7f16f415f500 2024-05-09T05:00:59.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16fc114640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f16f40735b0 con 0x7f16f415f8e0 2024-05-09T05:00:59.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16fa6b1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16f415fd40 0x7f16f40d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:59.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16fa6b1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16f415fd40 0x7f16f40d2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56044/0 (socket says 172.21.15.60:56044) 2024-05-09T05:00:59.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16fa6b1640 1 -- 172.21.15.60:0/1359761403 learned_addr learned my addr 172.21.15.60:0/1359761403 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:59.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16f96af640 1 --2- 172.21.15.60:0/1359761403 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f415f8e0 0x7f16f40ad470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:59.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16f9eb0640 1 --2- 172.21.15.60:0/1359761403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16f415f500 0x7f16f40b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:59.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16fa6b1640 1 -- 172.21.15.60:0/1359761403 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f415f8e0 msgr2=0x7f16f40ad470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:59.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16fa6b1640 1 --2- 172.21.15.60:0/1359761403 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f415f8e0 0x7f16f40ad470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16fa6b1640 1 -- 172.21.15.60:0/1359761403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16f415f500 msgr2=0x7f16f40b0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:59.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16fa6b1640 1 --2- 172.21.15.60:0/1359761403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16f415f500 0x7f16f40b0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16fa6b1640 1 -- 172.21.15.60:0/1359761403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f40adcc0 con 0x7f16f415fd40 2024-05-09T05:00:59.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16fa6b1640 1 --2- 172.21.15.60:0/1359761403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16f415fd40 0x7f16f40d2850 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f16ec034040 tx=0x7f16ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:59.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16ea7fc640 1 -- 172.21.15.60:0/1359761403 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec036050 con 0x7f16f415fd40 2024-05-09T05:00:59.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16ea7fc640 1 -- 172.21.15.60:0/1359761403 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16ec042070 con 0x7f16f415fd40 2024-05-09T05:00:59.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16fc114640 1 -- 172.21.15.60:0/1359761403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16f40adfa0 con 0x7f16f415fd40 2024-05-09T05:00:59.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.440+0000 7f16ea7fc640 1 -- 172.21.15.60:0/1359761403 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec009510 con 0x7f16f415fd40 2024-05-09T05:00:59.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.441+0000 7f16fc114640 1 -- 172.21.15.60:0/1359761403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16f40ae560 con 0x7f16f415fd40 2024-05-09T05:00:59.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.442+0000 7f16fc114640 1 -- 172.21.15.60:0/1359761403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16f4051040 con 0x7f16f415fd40 2024-05-09T05:00:59.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.442+0000 7f16ea7fc640 1 -- 172.21.15.60:0/1359761403 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f16ec007bf0 con 0x7f16f415fd40 2024-05-09T05:00:59.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.442+0000 7f16ea7fc640 1 --2- 172.21.15.60:0/1359761403 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f16d80762d0 0x7f16d8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:59.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.443+0000 7f16ea7fc640 1 -- 172.21.15.60:0/1359761403 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5249+0+0 (secure 0 0 0) 0x7f16ec0b8460 con 0x7f16f415fd40 2024-05-09T05:00:59.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.443+0000 7f16f9eb0640 1 --2- 172.21.15.60:0/1359761403 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f16d80762d0 0x7f16d8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:59.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.443+0000 7f16f9eb0640 1 --2- 172.21.15.60:0/1359761403 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f16d80762d0 0x7f16d8078790 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f16f0004ea0 tx=0x7f16f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:59.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.447+0000 7f16ea7fc640 1 -- 172.21.15.60:0/1359761403 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f16ec081ee0 con 0x7f16f415fd40 2024-05-09T05:00:59.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.709+0000 7f16fc114640 1 -- 172.21.15.60:0/1359761403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f16f409f2e0 con 0x7f16f415fd40 2024-05-09T05:00:59.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.710+0000 7f16ea7fc640 1 -- 172.21.15.60:0/1359761403 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) ==== 80+0+6645 (secure 0 0 0) 0x7f16ec0817e0 con 0x7f16f415fd40 2024-05-09T05:00:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:00:59.711 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:00:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2024-05-09T05:00:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2024-05-09T05:00:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAsWDxmJBQ1LRAAJfklX7rdjYuv6WIp+8rgMg==", 2024-05-09T05:00:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2024-05-09T05:00:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2024-05-09T05:00:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2024-05-09T05:00:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:00:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2024-05-09T05:00:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAsWDxmUN0fORAAaKZA9GlWlOfJaY/SYAXNlw==", 2024-05-09T05:00:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2024-05-09T05:00:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2024-05-09T05:00:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2024-05-09T05:00:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:00:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.c", 2024-05-09T05:00:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAsWDxmmr9iNBAAyatU9+hjHFV/AFTKpVxMPQ==", 2024-05-09T05:00:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2024-05-09T05:00:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2024-05-09T05:00:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2024-05-09T05:00:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:00:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.d", 2024-05-09T05:00:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAtWDxms0CIBBAAsHfza+C91qgfGjrczfoGeg==", 2024-05-09T05:00:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2024-05-09T05:00:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2024-05-09T05:00:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2024-05-09T05:00:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:00:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2024-05-09T05:00:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAuWDxmOInrKBAAthThp6oxwvbii2eif7krkg==", 2024-05-09T05:00:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2024-05-09T05:00:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2024-05-09T05:00:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:00:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2024-05-09T05:00:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAxWDxmbDifJxAAhJ5xGS3YfkNkYuJYF7MiIg==", 2024-05-09T05:00:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2024-05-09T05:00:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2024-05-09T05:00:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:00:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2024-05-09T05:00:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQA0WDxmrlOkJRAAf6eB9x9UcDCilfbWBIMCGQ==", 2024-05-09T05:00:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2024-05-09T05:00:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2024-05-09T05:00:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:00:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2024-05-09T05:00:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQA3WDxm2ItKJBAAO4FLP3sH/POba/k7DGFXqA==", 2024-05-09T05:00:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2024-05-09T05:00:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2024-05-09T05:00:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:00:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2024-05-09T05:00:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQA9WDxmKPPLKxAAGlyhDb222M4GpRybQ24Z3g==", 2024-05-09T05:00:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2024-05-09T05:00:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2024-05-09T05:00:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:00:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2024-05-09T05:00:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBAWDxmRKm6KRAAcZrQzTb6SQW5CQPyW9f+uQ==", 2024-05-09T05:00:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2024-05-09T05:00:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2024-05-09T05:00:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:00:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2024-05-09T05:00:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBDWDxmQkRwKBAA5/kHAwsRHh27fnYJuUh16A==", 2024-05-09T05:00:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2024-05-09T05:00:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2024-05-09T05:00:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:00:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2024-05-09T05:00:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBGWDxmAc98JhAAA/qroXPpF1fC79NGd7WN4w==", 2024-05-09T05:00:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2024-05-09T05:00:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2024-05-09T05:00:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:00:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2024-05-09T05:00:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAtWDxmiEa1CBAAm5ehYW6MVg0B6WUq8TxNbg==", 2024-05-09T05:00:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2024-05-09T05:00:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2024-05-09T05:00:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2024-05-09T05:00:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2024-05-09T05:00:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2024-05-09T05:00:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAtWDxmA/enERAABGFsfdUaqliUPbTu5oCWJQ==", 2024-05-09T05:00:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2024-05-09T05:00:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2024-05-09T05:00:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2024-05-09T05:00:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2024-05-09T05:00:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2024-05-09T05:00:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAtWDxmtWd4DhAAF1LGmIJOK3X1UoC2pTnOLw==", 2024-05-09T05:00:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2024-05-09T05:00:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2024-05-09T05:00:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2024-05-09T05:00:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2024-05-09T05:00:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.3", 2024-05-09T05:00:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAtWDxmFWFIFxAAY/B8R3xXgu+A509Ww5Hqjg==", 2024-05-09T05:00:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2024-05-09T05:00:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2024-05-09T05:00:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2024-05-09T05:00:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2024-05-09T05:00:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2024-05-09T05:00:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAsWDxmxY1wERAAjAXnXGh6+bKKDWkUu6TWJw==", 2024-05-09T05:00:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2024-05-09T05:00:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2024-05-09T05:00:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2024-05-09T05:00:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:00:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.bootstrap-mds", 2024-05-09T05:00:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBMWDxmGmHfHBAALyHGfcb9d4rg/EuxLBqMlA==", 2024-05-09T05:00:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile bootstrap-mds" 2024-05-09T05:00:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.bootstrap-mgr", 2024-05-09T05:00:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBMWDxmsmnfHBAAvakbhpu/sAAFFci5NM1u/w==", 2024-05-09T05:00:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile bootstrap-mgr" 2024-05-09T05:00:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.bootstrap-osd", 2024-05-09T05:00:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBMWDxmH3HfHBAAJM328ve+ERkrpnjPXpLptA==", 2024-05-09T05:00:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile bootstrap-osd" 2024-05-09T05:00:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.bootstrap-rbd", 2024-05-09T05:00:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBMWDxmIXjfHBAAmD/bbPTX2i8WhdFCgNwVlQ==", 2024-05-09T05:00:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile bootstrap-rbd" 2024-05-09T05:00:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-05-09T05:00:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBMWDxmmIDfHBAAXGwJmT5Rb0Cz3HSlgbC9Cg==", 2024-05-09T05:00:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-05-09T05:00:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.bootstrap-rgw", 2024-05-09T05:00:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBMWDxm7YffHBAA5Ev2FHmtjyqBedVGrnTsIA==", 2024-05-09T05:00:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile bootstrap-rgw" 2024-05-09T05:00:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2024-05-09T05:00:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAsWDxmuRPmKRAA8uT8sJ8VL1XVxa6c3RTj6g==", 2024-05-09T05:00:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2024-05-09T05:00:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2024-05-09T05:00:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:00:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:00:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:00:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2024-05-09T05:00:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAsWDxmdBNeJhAAyU0fz/hMt2qp285unqOtlQ==", 2024-05-09T05:00:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:00:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2024-05-09T05:00:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2024-05-09T05:00:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:00:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:00:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-05-09T05:00:59.727 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:00:59.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.712+0000 7f16fc114640 1 -- 172.21.15.60:0/1359761403 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f16d80762d0 msgr2=0x7f16d8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:59.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.712+0000 7f16fc114640 1 --2- 172.21.15.60:0/1359761403 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f16d80762d0 0x7f16d8078790 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f16f0004ea0 tx=0x7f16f0009040 comp rx=0 tx=0).stop 2024-05-09T05:00:59.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.712+0000 7f16fc114640 1 -- 172.21.15.60:0/1359761403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16f415fd40 msgr2=0x7f16f40d2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:59.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.712+0000 7f16fc114640 1 --2- 172.21.15.60:0/1359761403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16f415fd40 0x7f16f40d2850 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f16ec034040 tx=0x7f16ec004a40 comp rx=0 tx=0).stop 2024-05-09T05:00:59.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.712+0000 7f16fc114640 1 -- 172.21.15.60:0/1359761403 shutdown_connections 2024-05-09T05:00:59.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.712+0000 7f16fc114640 1 --2- 172.21.15.60:0/1359761403 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f16d80762d0 0x7f16d8078790 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.712+0000 7f16fc114640 1 --2- 172.21.15.60:0/1359761403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16f415fd40 0x7f16f40d2850 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.712+0000 7f16fc114640 1 --2- 172.21.15.60:0/1359761403 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f16f415f8e0 0x7f16f40ad470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.712+0000 7f16fc114640 1 --2- 172.21.15.60:0/1359761403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16f415f500 0x7f16f40b0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.712+0000 7f16fc114640 1 -- 172.21.15.60:0/1359761403 >> 172.21.15.60:0/1359761403 conn(0x7f16f40b4ec0 msgr2=0x7f16f4160300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:59.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.712+0000 7f16fc114640 1 -- 172.21.15.60:0/1359761403 shutdown_connections 2024-05-09T05:00:59.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.712+0000 7f16fc114640 1 -- 172.21.15.60:0/1359761403 wait complete. 2024-05-09T05:00:59.730 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-05-09T05:00:59.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.834+0000 7f7293b2b640 1 Processor -- start 2024-05-09T05:00:59.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.834+0000 7f7293b2b640 1 -- start start 2024-05-09T05:00:59.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.835+0000 7f7293b2b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f728c15f500 0x7f728c099b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:59.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.835+0000 7f7293b2b640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f728c09a040 0x7f728c15f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:59.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.835+0000 7f7293b2b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f728c09a4e0 con 0x7f728c15fe20 2024-05-09T05:00:59.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.835+0000 7f7293b2b640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f728c072ad0 con 0x7f728c09a040 2024-05-09T05:00:59.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.835+0000 7f7293b2b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f728c072db0 con 0x7f728c15f500 2024-05-09T05:00:59.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.835+0000 7f72920c8640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f728c15fe20 0x7f728c06fe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51454/0 (socket says 172.21.15.60:51454) 2024-05-09T05:00:59.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.835+0000 7f72920c8640 1 -- 172.21.15.60:0/533993038 learned_addr learned my addr 172.21.15.60:0/533993038 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:00:59.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72910c6640 1 --2- 172.21.15.60:0/533993038 >> v2:172.21.15.114:3300/0 conn(0x7f728c09a040 0x7f728c15f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:59.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72918c7640 1 --2- 172.21.15.60:0/533993038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f728c15f500 0x7f728c099b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:59.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72908c5640 1 -- 172.21.15.60:0/533993038 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4281167817 0 0) 0x7f728c09a4e0 con 0x7f728c15fe20 2024-05-09T05:00:59.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72908c5640 1 -- 172.21.15.60:0/533993038 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7274003520 con 0x7f728c15fe20 2024-05-09T05:00:59.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72908c5640 1 -- 172.21.15.60:0/533993038 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 216800756 0 0) 0x7f7274003520 con 0x7f728c15fe20 2024-05-09T05:00:59.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72908c5640 1 -- 172.21.15.60:0/533993038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f728c15f500 msgr2=0x7f728c099b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:59.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72908c5640 1 --2- 172.21.15.60:0/533993038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f728c15f500 0x7f728c099b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72908c5640 1 -- 172.21.15.60:0/533993038 >> v2:172.21.15.114:3300/0 conn(0x7f728c09a040 msgr2=0x7f728c15f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:59.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72908c5640 1 --2- 172.21.15.60:0/533993038 >> v2:172.21.15.114:3300/0 conn(0x7f728c09a040 0x7f728c15f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72908c5640 1 -- 172.21.15.60:0/533993038 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f728c09a7c0 con 0x7f728c15fe20 2024-05-09T05:00:59.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72908c5640 1 -- 172.21.15.60:0/533993038 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f727c001a70 con 0x7f728c15fe20 2024-05-09T05:00:59.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72908c5640 1 -- 172.21.15.60:0/533993038 >> v1:172.21.15.60:6789/0 conn(0x7f728c15fe20 legacy=0x7f728c06fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:59.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72908c5640 1 --2- 172.21.15.60:0/533993038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7274003a00 0x7f7274003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72908c5640 1 --2- 172.21.15.60:0/533993038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f728c15f500 0x7f7274004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72908c5640 1 --2- 172.21.15.60:0/533993038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7274004d40 0x7f72740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72908c5640 1 -- 172.21.15.60:0/533993038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f728c072ad0 con 0x7f7274004d40 2024-05-09T05:00:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72908c5640 1 -- 172.21.15.60:0/533993038 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f728c072db0 con 0x7f7274003a00 2024-05-09T05:00:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72908c5640 1 -- 172.21.15.60:0/533993038 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f727400ba40 con 0x7f728c15f500 2024-05-09T05:00:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72910c6640 1 --2- 172.21.15.60:0/533993038 >> v2:172.21.15.114:3300/0 conn(0x7f728c09a040 0x7f728c15f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:00:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72920c8640 1 --2- 172.21.15.60:0/533993038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7274004d40 0x7f72740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72918c7640 1 --2- 172.21.15.60:0/533993038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7274003a00 0x7f7274003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.836+0000 7f72910c6640 1 --2- 172.21.15.60:0/533993038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f728c15f500 0x7f7274004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.837+0000 7f72920c8640 1 -- 172.21.15.60:0/533993038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f728c15f500 msgr2=0x7f7274004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.837+0000 7f72920c8640 1 --2- 172.21.15.60:0/533993038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f728c15f500 0x7f7274004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.837+0000 7f72920c8640 1 -- 172.21.15.60:0/533993038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7274003a00 msgr2=0x7f7274003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:00:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.837+0000 7f72920c8640 1 --2- 172.21.15.60:0/533993038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7274003a00 0x7f7274003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.837+0000 7f72920c8640 1 -- 172.21.15.60:0/533993038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7274003520 con 0x7f7274004d40 2024-05-09T05:00:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.837+0000 7f72918c7640 1 --2- 172.21.15.60:0/533993038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7274003a00 0x7f7274003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.837+0000 7f72910c6640 1 --2- 172.21.15.60:0/533993038 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f728c15f500 0x7f7274004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:00:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.837+0000 7f72920c8640 1 --2- 172.21.15.60:0/533993038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7274004d40 0x7f72740051c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f728c099780 tx=0x7f7274005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c991fdbdf90ef22c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.837+0000 7f72908c5640 1 -- 172.21.15.60:0/533993038 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c00a4d0 con 0x7f7274004d40 2024-05-09T05:00:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.837+0000 7f72908c5640 1 -- 172.21.15.60:0/533993038 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f727c02f070 con 0x7f7274004d40 2024-05-09T05:00:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.837+0000 7f72908c5640 1 -- 172.21.15.60:0/533993038 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c02bbd0 con 0x7f7274004d40 2024-05-09T05:00:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.838+0000 7f7293b2b640 1 -- 172.21.15.60:0/533993038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7274004d40 msgr2=0x7f72740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.838+0000 7f7293b2b640 1 --2- 172.21.15.60:0/533993038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7274004d40 0x7f72740051c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f728c099780 tx=0x7f7274005830 comp rx=0 tx=0).stop 2024-05-09T05:00:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.838+0000 7f72920c8640 1 -- 172.21.15.60:0/533993038 reap_dead start 2024-05-09T05:00:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.838+0000 7f7293b2b640 1 -- 172.21.15.60:0/533993038 shutdown_connections 2024-05-09T05:00:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.838+0000 7f7293b2b640 1 -- 172.21.15.60:0/533993038 >> 172.21.15.60:0/533993038 conn(0x7f728c0b4ec0 msgr2=0x7f728c06e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:00:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.838+0000 7f7293b2b640 1 -- 172.21.15.60:0/533993038 shutdown_connections 2024-05-09T05:00:59.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.838+0000 7f7293b2b640 1 -- 172.21.15.60:0/533993038 wait complete. 2024-05-09T05:00:59.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.839+0000 7f7293b2b640 1 Processor -- start 2024-05-09T05:00:59.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.839+0000 7f7293b2b640 1 -- start start 2024-05-09T05:00:59.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.840+0000 7f7293b2b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7274004d40 0x7f728c160380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:59.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.840+0000 7f7293b2b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728c0ae5b0 0x7f728c1f5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:59.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.840+0000 7f7293b2b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f728c15a940 0x7f728c0af2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:59.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.840+0000 7f7293b2b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f728c070530 con 0x7f728c15a940 2024-05-09T05:00:59.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.840+0000 7f7293b2b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f728c09c1d0 con 0x7f728c0ae5b0 2024-05-09T05:00:59.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.840+0000 7f7293b2b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f728c0706b0 con 0x7f7274004d40 2024-05-09T05:00:59.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.840+0000 7f72920c8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f728c15a940 0x7f728c0af2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:59.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.840+0000 7f72920c8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f728c15a940 0x7f728c0af2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56066/0 (socket says 172.21.15.60:56066) 2024-05-09T05:00:59.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.840+0000 7f72920c8640 1 -- 172.21.15.60:0/3126710594 learned_addr learned my addr 172.21.15.60:0/3126710594 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:00:59.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.841+0000 7f72918c7640 1 --2- 172.21.15.60:0/3126710594 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7274004d40 0x7f728c160380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:59.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.841+0000 7f72910c6640 1 --2- 172.21.15.60:0/3126710594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728c0ae5b0 0x7f728c1f5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:59.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.841+0000 7f72920c8640 1 -- 172.21.15.60:0/3126710594 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7274004d40 msgr2=0x7f728c160380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:59.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.841+0000 7f72920c8640 1 --2- 172.21.15.60:0/3126710594 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7274004d40 0x7f728c160380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.841+0000 7f72920c8640 1 -- 172.21.15.60:0/3126710594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728c0ae5b0 msgr2=0x7f728c1f5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:00:59.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.841+0000 7f72920c8640 1 --2- 172.21.15.60:0/3126710594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728c0ae5b0 0x7f728c1f5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:00:59.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.841+0000 7f72920c8640 1 -- 172.21.15.60:0/3126710594 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f728c1f5c70 con 0x7f728c15a940 2024-05-09T05:00:59.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.841+0000 7f72910c6640 1 --2- 172.21.15.60:0/3126710594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728c0ae5b0 0x7f728c1f5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:00:59.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.841+0000 7f72920c8640 1 --2- 172.21.15.60:0/3126710594 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f728c15a940 0x7f728c0af2a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f727c00a940 tx=0x7f727c001db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:59.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.841+0000 7f728a7fc640 1 -- 172.21.15.60:0/3126710594 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c002020 con 0x7f728c15a940 2024-05-09T05:00:59.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.841+0000 7f728a7fc640 1 -- 172.21.15.60:0/3126710594 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f727c001a60 con 0x7f728c15a940 2024-05-09T05:00:59.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.841+0000 7f7293b2b640 1 -- 172.21.15.60:0/3126710594 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f728c0ad510 con 0x7f728c15a940 2024-05-09T05:00:59.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.841+0000 7f728a7fc640 1 -- 172.21.15.60:0/3126710594 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c02b9c0 con 0x7f728c15a940 2024-05-09T05:00:59.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.841+0000 7f7293b2b640 1 -- 172.21.15.60:0/3126710594 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f728c0adad0 con 0x7f728c15a940 2024-05-09T05:00:59.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.842+0000 7f7293b2b640 1 -- 172.21.15.60:0/3126710594 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f728c051040 con 0x7f728c15a940 2024-05-09T05:00:59.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.843+0000 7f728a7fc640 1 -- 172.21.15.60:0/3126710594 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f727c03e030 con 0x7f728c15a940 2024-05-09T05:00:59.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.843+0000 7f728a7fc640 1 --2- 172.21.15.60:0/3126710594 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f72780762d0 0x7f7278078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:00:59.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.843+0000 7f728a7fc640 1 -- 172.21.15.60:0/3126710594 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5249+0+0 (secure 0 0 0) 0x7f727c0b6b60 con 0x7f728c15a940 2024-05-09T05:00:59.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.843+0000 7f72918c7640 1 --2- 172.21.15.60:0/3126710594 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f72780762d0 0x7f7278078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:00:59.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.844+0000 7f72918c7640 1 --2- 172.21.15.60:0/3126710594 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f72780762d0 0x7f7278078790 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7284009990 tx=0x7f7284008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:00:59.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:00:59.848+0000 7f728a7fc640 1 -- 172.21.15.60:0/3126710594 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f728c1f5c70 con 0x7f728c15a940 2024-05-09T05:01:00.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.111+0000 7f7293b2b640 1 -- 172.21.15.60:0/3126710594 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f728c0701d0 con 0x7f728c15a940 2024-05-09T05:01:00.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.113+0000 7f728a7fc640 1 -- 172.21.15.60:0/3126710594 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f727c0805f0 con 0x7f728c15a940 2024-05-09T05:01:00.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.115+0000 7f7293b2b640 1 -- 172.21.15.60:0/3126710594 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f72780762d0 msgr2=0x7f7278078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:00.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.115+0000 7f7293b2b640 1 --2- 172.21.15.60:0/3126710594 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f72780762d0 0x7f7278078790 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7284009990 tx=0x7f7284008040 comp rx=0 tx=0).stop 2024-05-09T05:01:00.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.115+0000 7f7293b2b640 1 -- 172.21.15.60:0/3126710594 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f728c15a940 msgr2=0x7f728c0af2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:00.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.115+0000 7f7293b2b640 1 --2- 172.21.15.60:0/3126710594 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f728c15a940 0x7f728c0af2a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f727c00a940 tx=0x7f727c001db0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.115+0000 7f7293b2b640 1 -- 172.21.15.60:0/3126710594 shutdown_connections 2024-05-09T05:01:00.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.115+0000 7f7293b2b640 1 --2- 172.21.15.60:0/3126710594 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f72780762d0 0x7f7278078790 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.115+0000 7f7293b2b640 1 --2- 172.21.15.60:0/3126710594 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f728c15a940 0x7f728c0af2a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.115+0000 7f7293b2b640 1 --2- 172.21.15.60:0/3126710594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728c0ae5b0 0x7f728c1f5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.115+0000 7f7293b2b640 1 --2- 172.21.15.60:0/3126710594 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7274004d40 0x7f728c160380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.115+0000 7f7293b2b640 1 -- 172.21.15.60:0/3126710594 >> 172.21.15.60:0/3126710594 conn(0x7f728c0b4ec0 msgr2=0x7f728c09d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:00.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.115+0000 7f7293b2b640 1 -- 172.21.15.60:0/3126710594 shutdown_connections 2024-05-09T05:01:00.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.115+0000 7f7293b2b640 1 -- 172.21.15.60:0/3126710594 wait complete. 2024-05-09T05:01:00.126 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-05-09T05:01:00.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.230+0000 7fd87bac8640 1 Processor -- start 2024-05-09T05:01:00.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.230+0000 7fd87bac8640 1 -- start start 2024-05-09T05:01:00.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.231+0000 7fd87bac8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8740ae150 0x7fd87409c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:00.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.231+0000 7fd87bac8640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8740b2550 con 0x7fd8740b16e0 2024-05-09T05:01:00.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.231+0000 7fd87bac8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8740ab4b0 con 0x7fd87409cba0 2024-05-09T05:01:00.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.231+0000 7fd87bac8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd87409d2a0 con 0x7fd8740ae150 2024-05-09T05:01:00.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.231+0000 7fd87a065640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd8740b16e0 0x7fd8740b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51462/0 (socket says 172.21.15.60:51462) 2024-05-09T05:01:00.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.231+0000 7fd87a065640 1 -- 172.21.15.60:0/1111268005 learned_addr learned my addr 172.21.15.60:0/1111268005 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:00.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.231+0000 7fd878862640 1 -- 172.21.15.60:0/1111268005 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4003498473 0 0) 0x7fd8740b2550 con 0x7fd8740b16e0 2024-05-09T05:01:00.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.231+0000 7fd879864640 1 --2- 172.21.15.60:0/1111268005 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8740ae150 0x7fd87409c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:00.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.231+0000 7fd878862640 1 -- 172.21.15.60:0/1111268005 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd860003520 con 0x7fd8740b16e0 2024-05-09T05:01:00.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd878862640 1 -- 172.21.15.60:0/1111268005 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 317011145 0 0) 0x7fd860003520 con 0x7fd8740b16e0 2024-05-09T05:01:00.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd878862640 1 -- 172.21.15.60:0/1111268005 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8740ae150 msgr2=0x7fd87409c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:00.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd878862640 1 --2- 172.21.15.60:0/1111268005 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8740ae150 0x7fd87409c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd878862640 1 -- 172.21.15.60:0/1111268005 >> v1:172.21.15.114:6789/0 conn(0x7fd87409cba0 legacy=0x7fd8740b0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:00.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd878862640 1 -- 172.21.15.60:0/1111268005 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87409d580 con 0x7fd8740b16e0 2024-05-09T05:01:00.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd878862640 1 -- 172.21.15.60:0/1111268005 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7fd86c001a70 con 0x7fd8740b16e0 2024-05-09T05:01:00.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd878862640 1 -- 172.21.15.60:0/1111268005 >> v1:172.21.15.60:6789/0 conn(0x7fd8740b16e0 legacy=0x7fd8740b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:00.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd878862640 1 --2- 172.21.15.60:0/1111268005 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8740ae150 0x7fd860003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:00.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd878862640 1 --2- 172.21.15.60:0/1111268005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd860004000 0x7fd860004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:00.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd878862640 1 --2- 172.21.15.60:0/1111268005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd860004d40 0x7fd8600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:00.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd878862640 1 -- 172.21.15.60:0/1111268005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd87409d2a0 con 0x7fd860004d40 2024-05-09T05:01:00.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd878862640 1 -- 172.21.15.60:0/1111268005 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd86000bb50 con 0x7fd860004000 2024-05-09T05:01:00.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd878862640 1 -- 172.21.15.60:0/1111268005 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd86000bd50 con 0x7fd8740ae150 2024-05-09T05:01:00.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd87a065640 1 --2- 172.21.15.60:0/1111268005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd860004d40 0x7fd8600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:00.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd879063640 1 --2- 172.21.15.60:0/1111268005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd860004000 0x7fd860004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:00.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd879864640 1 --2- 172.21.15.60:0/1111268005 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8740ae150 0x7fd860003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:00.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd87a065640 1 -- 172.21.15.60:0/1111268005 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8740ae150 msgr2=0x7fd860003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:00.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd87a065640 1 --2- 172.21.15.60:0/1111268005 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8740ae150 0x7fd860003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd87a065640 1 -- 172.21.15.60:0/1111268005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd860004000 msgr2=0x7fd860004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:00.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd87a065640 1 --2- 172.21.15.60:0/1111268005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd860004000 0x7fd860004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.232+0000 7fd87a065640 1 -- 172.21.15.60:0/1111268005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd860003520 con 0x7fd860004d40 2024-05-09T05:01:00.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.233+0000 7fd87a065640 1 --2- 172.21.15.60:0/1111268005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd860004d40 0x7fd8600051c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00a830 tx=0x7fd860005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37778df3b633a0ba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:00.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.233+0000 7fd878862640 1 -- 172.21.15.60:0/1111268005 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86c00a450 con 0x7fd860004d40 2024-05-09T05:01:00.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.233+0000 7fd878862640 1 -- 172.21.15.60:0/1111268005 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd86c02f070 con 0x7fd860004d40 2024-05-09T05:01:00.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.233+0000 7fd878862640 1 -- 172.21.15.60:0/1111268005 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86c02bc00 con 0x7fd860004d40 2024-05-09T05:01:00.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.234+0000 7fd87bac8640 1 -- 172.21.15.60:0/1111268005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd860004d40 msgr2=0x7fd8600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:00.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.234+0000 7fd87bac8640 1 --2- 172.21.15.60:0/1111268005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd860004d40 0x7fd8600051c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00a830 tx=0x7fd860005830 comp rx=0 tx=0).stop 2024-05-09T05:01:00.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.234+0000 7fd87a065640 1 -- 172.21.15.60:0/1111268005 reap_dead start 2024-05-09T05:01:00.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.234+0000 7fd87bac8640 1 -- 172.21.15.60:0/1111268005 shutdown_connections 2024-05-09T05:01:00.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.234+0000 7fd87bac8640 1 -- 172.21.15.60:0/1111268005 >> 172.21.15.60:0/1111268005 conn(0x7fd874072390 msgr2=0x7fd8740727a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:00.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.234+0000 7fd87bac8640 1 -- 172.21.15.60:0/1111268005 shutdown_connections 2024-05-09T05:01:00.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.234+0000 7fd87bac8640 1 -- 172.21.15.60:0/1111268005 wait complete. 2024-05-09T05:01:00.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.235+0000 7fd87bac8640 1 Processor -- start 2024-05-09T05:01:00.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.235+0000 7fd87bac8640 1 -- start start 2024-05-09T05:01:00.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd87bac8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd860004d40 0x7fd874162310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:00.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd87bac8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8740d07d0 0x7fd87406fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:00.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd87bac8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8740d0c30 0x7fd87414b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:00.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd87bac8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd87409b730 con 0x7fd8740d0c30 2024-05-09T05:01:00.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd87bac8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd87409d860 con 0x7fd860004d40 2024-05-09T05:01:00.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd87bac8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd87406e970 con 0x7fd8740d07d0 2024-05-09T05:01:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd87a065640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8740d0c30 0x7fd87414b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd87a065640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8740d0c30 0x7fd87414b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56084/0 (socket says 172.21.15.60:56084) 2024-05-09T05:01:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd87a065640 1 -- 172.21.15.60:0/2544832949 learned_addr learned my addr 172.21.15.60:0/2544832949 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd879864640 1 --2- 172.21.15.60:0/2544832949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd860004d40 0x7fd874162310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd879063640 1 --2- 172.21.15.60:0/2544832949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8740d07d0 0x7fd87406fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd87a065640 1 -- 172.21.15.60:0/2544832949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8740d07d0 msgr2=0x7fd87406fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd87a065640 1 --2- 172.21.15.60:0/2544832949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8740d07d0 0x7fd87406fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd87a065640 1 -- 172.21.15.60:0/2544832949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd860004d40 msgr2=0x7fd874162310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd87a065640 1 --2- 172.21.15.60:0/2544832949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd860004d40 0x7fd874162310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd87a065640 1 -- 172.21.15.60:0/2544832949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87406ec50 con 0x7fd8740d0c30 2024-05-09T05:01:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd879864640 1 --2- 172.21.15.60:0/2544832949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd860004d40 0x7fd874162310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:00.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd87a065640 1 --2- 172.21.15.60:0/2544832949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8740d0c30 0x7fd87414b4f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00a950 tx=0x7fd8740b1b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd86a7fc640 1 -- 172.21.15.60:0/2544832949 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86c00a450 con 0x7fd8740d0c30 2024-05-09T05:01:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd86a7fc640 1 -- 172.21.15.60:0/2544832949 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd86c004b60 con 0x7fd8740d0c30 2024-05-09T05:01:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd87bac8640 1 -- 172.21.15.60:0/2544832949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd87406ef30 con 0x7fd8740d0c30 2024-05-09T05:01:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd86a7fc640 1 -- 172.21.15.60:0/2544832949 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86c001fa0 con 0x7fd8740d0c30 2024-05-09T05:01:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.236+0000 7fd87bac8640 1 -- 172.21.15.60:0/2544832949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd874162850 con 0x7fd8740d0c30 2024-05-09T05:01:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.238+0000 7fd87bac8640 1 -- 172.21.15.60:0/2544832949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd874051040 con 0x7fd8740d0c30 2024-05-09T05:01:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.238+0000 7fd86a7fc640 1 -- 172.21.15.60:0/2544832949 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd86c034470 con 0x7fd8740d0c30 2024-05-09T05:01:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.239+0000 7fd86a7fc640 1 --2- 172.21.15.60:0/2544832949 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd8580762d0 0x7fd858078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.239+0000 7fd86a7fc640 1 -- 172.21.15.60:0/2544832949 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5249+0+0 (secure 0 0 0) 0x7fd86c0b6c20 con 0x7fd8740d0c30 2024-05-09T05:01:00.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.239+0000 7fd879864640 1 --2- 172.21.15.60:0/2544832949 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd8580762d0 0x7fd858078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:00.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.239+0000 7fd879864640 1 --2- 172.21.15.60:0/2544832949 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd8580762d0 0x7fd858078790 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd870009bc0 tx=0x7fd870008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:00.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.243+0000 7fd86a7fc640 1 -- 172.21.15.60:0/2544832949 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd86c0807d0 con 0x7fd8740d0c30 2024-05-09T05:01:00.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.515+0000 7fd87bac8640 1 -- 172.21.15.60:0/2544832949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7fd87409af40 con 0x7fd8740d0c30 2024-05-09T05:01:00.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.517+0000 7fd86a7fc640 1 -- 172.21.15.60:0/2544832949 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7fd86c0800d0 con 0x7fd8740d0c30 2024-05-09T05:01:00.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.520+0000 7fd87bac8640 1 -- 172.21.15.60:0/2544832949 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd8580762d0 msgr2=0x7fd858078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:00.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.520+0000 7fd87bac8640 1 --2- 172.21.15.60:0/2544832949 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd8580762d0 0x7fd858078790 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd870009bc0 tx=0x7fd870008040 comp rx=0 tx=0).stop 2024-05-09T05:01:00.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.520+0000 7fd87bac8640 1 -- 172.21.15.60:0/2544832949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8740d0c30 msgr2=0x7fd87414b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:00.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.520+0000 7fd87bac8640 1 --2- 172.21.15.60:0/2544832949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8740d0c30 0x7fd87414b4f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00a950 tx=0x7fd8740b1b90 comp rx=0 tx=0).stop 2024-05-09T05:01:00.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.520+0000 7fd87bac8640 1 -- 172.21.15.60:0/2544832949 shutdown_connections 2024-05-09T05:01:00.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.520+0000 7fd87bac8640 1 --2- 172.21.15.60:0/2544832949 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd8580762d0 0x7fd858078790 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.520+0000 7fd87bac8640 1 --2- 172.21.15.60:0/2544832949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8740d0c30 0x7fd87414b4f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.520+0000 7fd87bac8640 1 --2- 172.21.15.60:0/2544832949 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd8740d07d0 0x7fd87406fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.520+0000 7fd87bac8640 1 --2- 172.21.15.60:0/2544832949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd860004d40 0x7fd874162310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.520+0000 7fd87bac8640 1 -- 172.21.15.60:0/2544832949 >> 172.21.15.60:0/2544832949 conn(0x7fd874072390 msgr2=0x7fd874099600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:00.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.520+0000 7fd87bac8640 1 -- 172.21.15.60:0/2544832949 shutdown_connections 2024-05-09T05:01:00.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.520+0000 7fd87bac8640 1 -- 172.21.15.60:0/2544832949 wait complete. 2024-05-09T05:01:00.531 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-05-09T05:01:00.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.635+0000 7f5aea499640 1 Processor -- start 2024-05-09T05:01:00.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.635+0000 7f5aea499640 1 -- start start 2024-05-09T05:01:00.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.636+0000 7f5aea499640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae41606f0 0x7f5ae40d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:00.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.636+0000 7f5aea499640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5ae406f9f0 0x7f5ae40d3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:00.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.636+0000 7f5aea499640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ae406ff60 con 0x7f5ae415f300 2024-05-09T05:01:00.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.636+0000 7f5aea499640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f5ae4070240 con 0x7f5ae406f9f0 2024-05-09T05:01:00.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.636+0000 7f5aea499640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f5ae4070520 con 0x7f5ae41606f0 2024-05-09T05:01:00.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.636+0000 7f5ae3fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5ae415f300 0x7f5ae41639f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51476/0 (socket says 172.21.15.60:51476) 2024-05-09T05:01:00.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.636+0000 7f5ae3fff640 1 -- 172.21.15.60:0/1611729808 learned_addr learned my addr 172.21.15.60:0/1611729808 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:00.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae8a36640 1 --2- 172.21.15.60:0/1611729808 >> v2:172.21.15.114:3300/0 conn(0x7f5ae406f9f0 0x7f5ae40d3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:00.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae37fe640 1 --2- 172.21.15.60:0/1611729808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae41606f0 0x7f5ae40d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:00.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae2ffd640 1 -- 172.21.15.60:0/1611729808 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3780490362 0 0) 0x7f5ae406ff60 con 0x7f5ae415f300 2024-05-09T05:01:00.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae2ffd640 1 -- 172.21.15.60:0/1611729808 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5acc003520 con 0x7f5ae415f300 2024-05-09T05:01:00.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae2ffd640 1 -- 172.21.15.60:0/1611729808 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1362521829 0 0) 0x7f5acc003520 con 0x7f5ae415f300 2024-05-09T05:01:00.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae2ffd640 1 -- 172.21.15.60:0/1611729808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae41606f0 msgr2=0x7f5ae40d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:00.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae2ffd640 1 --2- 172.21.15.60:0/1611729808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae41606f0 0x7f5ae40d33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae2ffd640 1 -- 172.21.15.60:0/1611729808 >> v2:172.21.15.114:3300/0 conn(0x7f5ae406f9f0 msgr2=0x7f5ae40d3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:00.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae2ffd640 1 --2- 172.21.15.60:0/1611729808 >> v2:172.21.15.114:3300/0 conn(0x7f5ae406f9f0 0x7f5ae40d3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae2ffd640 1 -- 172.21.15.60:0/1611729808 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae406e740 con 0x7f5ae415f300 2024-05-09T05:01:00.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae2ffd640 1 -- 172.21.15.60:0/1611729808 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f5ad40016f0 con 0x7f5ae415f300 2024-05-09T05:01:00.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae2ffd640 1 -- 172.21.15.60:0/1611729808 >> v1:172.21.15.60:6789/0 conn(0x7f5ae415f300 legacy=0x7f5ae41639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:00.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae2ffd640 1 --2- 172.21.15.60:0/1611729808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5acc003a00 0x7f5acc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:00.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae2ffd640 1 --2- 172.21.15.60:0/1611729808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae41606f0 0x7f5acc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:00.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae2ffd640 1 --2- 172.21.15.60:0/1611729808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5acc004d40 0x7f5acc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:00.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae2ffd640 1 -- 172.21.15.60:0/1611729808 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae4070240 con 0x7f5acc004d40 2024-05-09T05:01:00.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae2ffd640 1 -- 172.21.15.60:0/1611729808 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae4070520 con 0x7f5acc003a00 2024-05-09T05:01:00.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae2ffd640 1 -- 172.21.15.60:0/1611729808 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f5acc00b9f0 con 0x7f5ae41606f0 2024-05-09T05:01:00.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae8a36640 1 --2- 172.21.15.60:0/1611729808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5acc004d40 0x7f5acc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:00.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae37fe640 1 --2- 172.21.15.60:0/1611729808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae41606f0 0x7f5acc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:00.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae3fff640 1 --2- 172.21.15.60:0/1611729808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5acc003a00 0x7f5acc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:00.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae8a36640 1 -- 172.21.15.60:0/1611729808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae41606f0 msgr2=0x7f5acc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:00.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae8a36640 1 --2- 172.21.15.60:0/1611729808 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae41606f0 0x7f5acc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae8a36640 1 -- 172.21.15.60:0/1611729808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5acc003a00 msgr2=0x7f5acc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:00.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae8a36640 1 --2- 172.21.15.60:0/1611729808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5acc003a00 0x7f5acc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.637+0000 7f5ae8a36640 1 -- 172.21.15.60:0/1611729808 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5acc003520 con 0x7f5acc004d40 2024-05-09T05:01:00.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.638+0000 7f5ae8a36640 1 --2- 172.21.15.60:0/1611729808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5acc004d40 0x7f5acc0051c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5adc009fc0 tx=0x7f5acc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae301c2ef88bc61 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:00.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.638+0000 7f5ae2ffd640 1 -- 172.21.15.60:0/1611729808 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5adc009c10 con 0x7f5acc004d40 2024-05-09T05:01:00.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.638+0000 7f5ae2ffd640 1 -- 172.21.15.60:0/1611729808 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5adc036070 con 0x7f5acc004d40 2024-05-09T05:01:00.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.638+0000 7f5ae2ffd640 1 -- 172.21.15.60:0/1611729808 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5adc02d070 con 0x7f5acc004d40 2024-05-09T05:01:00.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.639+0000 7f5aea499640 1 -- 172.21.15.60:0/1611729808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5acc004d40 msgr2=0x7f5acc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:00.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.639+0000 7f5aea499640 1 --2- 172.21.15.60:0/1611729808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5acc004d40 0x7f5acc0051c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5adc009fc0 tx=0x7f5acc005830 comp rx=0 tx=0).stop 2024-05-09T05:01:00.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.639+0000 7f5ae8a36640 1 -- 172.21.15.60:0/1611729808 reap_dead start 2024-05-09T05:01:00.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.639+0000 7f5aea499640 1 -- 172.21.15.60:0/1611729808 shutdown_connections 2024-05-09T05:01:00.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.639+0000 7f5aea499640 1 -- 172.21.15.60:0/1611729808 >> 172.21.15.60:0/1611729808 conn(0x7f5ae409b850 msgr2=0x7f5ae409bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:00.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.639+0000 7f5aea499640 1 -- 172.21.15.60:0/1611729808 shutdown_connections 2024-05-09T05:01:00.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.639+0000 7f5aea499640 1 -- 172.21.15.60:0/1611729808 wait complete. 2024-05-09T05:01:00.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.640+0000 7f5aea499640 1 Processor -- start 2024-05-09T05:01:00.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.640+0000 7f5aea499640 1 -- start start 2024-05-09T05:01:00.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.641+0000 7f5aea499640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5acc004d40 0x7f5ae41f8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:00.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.641+0000 7f5aea499640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae41f21d0 0x7f5ae41f2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:00.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.641+0000 7f5aea499640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae40ae3b0 0x7f5ae40ae830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:00.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.641+0000 7f5aea499640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae40d3570 con 0x7f5acc004d40 2024-05-09T05:01:00.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.641+0000 7f5aea499640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae40d33f0 con 0x7f5ae41f21d0 2024-05-09T05:01:00.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.641+0000 7f5aea499640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f5ae40d36f0 con 0x7f5ae40ae3b0 2024-05-09T05:01:00.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.641+0000 7f5ae3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5acc004d40 0x7f5ae41f8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:00.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.641+0000 7f5ae3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5acc004d40 0x7f5ae41f8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56112/0 (socket says 172.21.15.60:56112) 2024-05-09T05:01:00.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.641+0000 7f5ae3fff640 1 -- 172.21.15.60:0/615693301 learned_addr learned my addr 172.21.15.60:0/615693301 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:00.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.641+0000 7f5ae37fe640 1 --2- 172.21.15.60:0/615693301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae41f21d0 0x7f5ae41f2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:00.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.641+0000 7f5ae8a36640 1 --2- 172.21.15.60:0/615693301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae40ae3b0 0x7f5ae40ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:00.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.641+0000 7f5ae3fff640 1 -- 172.21.15.60:0/615693301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae40ae3b0 msgr2=0x7f5ae40ae830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:00.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.641+0000 7f5ae3fff640 1 --2- 172.21.15.60:0/615693301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae40ae3b0 0x7f5ae40ae830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.641+0000 7f5ae3fff640 1 -- 172.21.15.60:0/615693301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae41f21d0 msgr2=0x7f5ae41f2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:00.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.641+0000 7f5ae3fff640 1 --2- 172.21.15.60:0/615693301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae41f21d0 0x7f5ae41f2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.641+0000 7f5ae3fff640 1 -- 172.21.15.60:0/615693301 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae40af150 con 0x7f5acc004d40 2024-05-09T05:01:00.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.641+0000 7f5ae8a36640 1 --2- 172.21.15.60:0/615693301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae40ae3b0 0x7f5ae40ae830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:00.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.641+0000 7f5ae37fe640 1 --2- 172.21.15.60:0/615693301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae41f21d0 0x7f5ae41f2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:00.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.642+0000 7f5ae3fff640 1 --2- 172.21.15.60:0/615693301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5acc004d40 0x7f5ae41f8a00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4007340 tx=0x7f5ad4007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:00.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.642+0000 7f5ae0ff9640 1 -- 172.21.15.60:0/615693301 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad400f860 con 0x7f5acc004d40 2024-05-09T05:01:00.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.642+0000 7f5ae0ff9640 1 -- 172.21.15.60:0/615693301 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ad4018070 con 0x7f5acc004d40 2024-05-09T05:01:00.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.642+0000 7f5aea499640 1 -- 172.21.15.60:0/615693301 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ae41f65b0 con 0x7f5acc004d40 2024-05-09T05:01:00.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.642+0000 7f5ae0ff9640 1 -- 172.21.15.60:0/615693301 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad400fc60 con 0x7f5acc004d40 2024-05-09T05:01:00.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.642+0000 7f5aea499640 1 -- 172.21.15.60:0/615693301 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ae41f6b70 con 0x7f5acc004d40 2024-05-09T05:01:00.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.643+0000 7f5aea499640 1 -- 172.21.15.60:0/615693301 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ae41f6d70 con 0x7f5acc004d40 2024-05-09T05:01:00.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.643+0000 7f5ae0ff9640 1 -- 172.21.15.60:0/615693301 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5ad401d460 con 0x7f5acc004d40 2024-05-09T05:01:00.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.644+0000 7f5ae0ff9640 1 --2- 172.21.15.60:0/615693301 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5ad00762d0 0x7f5ad0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:00.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.644+0000 7f5ae0ff9640 1 -- 172.21.15.60:0/615693301 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5249+0+0 (secure 0 0 0) 0x7f5ad40973e0 con 0x7f5acc004d40 2024-05-09T05:01:00.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.644+0000 7f5ae37fe640 1 --2- 172.21.15.60:0/615693301 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5ad00762d0 0x7f5ad0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:00.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.645+0000 7f5ae37fe640 1 --2- 172.21.15.60:0/615693301 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5ad00762d0 0x7f5ad0078790 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5ae40b0470 tx=0x7f5ad8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:00.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.649+0000 7f5ae0ff9640 1 -- 172.21.15.60:0/615693301 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5ae41f6d70 con 0x7f5acc004d40 2024-05-09T05:01:00.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.914+0000 7f5aea499640 1 -- 172.21.15.60:0/615693301 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f5ae4051040 con 0x7f5acc004d40 2024-05-09T05:01:00.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.916+0000 7f5ae0ff9640 1 -- 172.21.15.60:0/615693301 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) ==== 90+0+0 (secure 0 0 0) 0x7f5ad4060f90 con 0x7f5acc004d40 2024-05-09T05:01:00.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.918+0000 7f5aea499640 1 -- 172.21.15.60:0/615693301 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5ad00762d0 msgr2=0x7f5ad0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:00.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.918+0000 7f5aea499640 1 --2- 172.21.15.60:0/615693301 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5ad00762d0 0x7f5ad0078790 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5ae40b0470 tx=0x7f5ad8008040 comp rx=0 tx=0).stop 2024-05-09T05:01:00.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.918+0000 7f5aea499640 1 -- 172.21.15.60:0/615693301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5acc004d40 msgr2=0x7f5ae41f8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:00.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.918+0000 7f5aea499640 1 --2- 172.21.15.60:0/615693301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5acc004d40 0x7f5ae41f8a00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4007340 tx=0x7f5ad4007800 comp rx=0 tx=0).stop 2024-05-09T05:01:00.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.918+0000 7f5aea499640 1 -- 172.21.15.60:0/615693301 shutdown_connections 2024-05-09T05:01:00.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.918+0000 7f5aea499640 1 --2- 172.21.15.60:0/615693301 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5ad00762d0 0x7f5ad0078790 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.918+0000 7f5aea499640 1 --2- 172.21.15.60:0/615693301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ae40ae3b0 0x7f5ae40ae830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.918+0000 7f5aea499640 1 --2- 172.21.15.60:0/615693301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ae41f21d0 0x7f5ae41f2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.918+0000 7f5aea499640 1 --2- 172.21.15.60:0/615693301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5acc004d40 0x7f5ae41f8a00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:00.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.918+0000 7f5aea499640 1 -- 172.21.15.60:0/615693301 >> 172.21.15.60:0/615693301 conn(0x7f5ae409b850 msgr2=0x7f5ae40d3900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:00.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.918+0000 7f5aea499640 1 -- 172.21.15.60:0/615693301 shutdown_connections 2024-05-09T05:01:00.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:00.918+0000 7f5aea499640 1 -- 172.21.15.60:0/615693301 wait complete. 2024-05-09T05:01:00.931 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-05-09T05:01:01.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.035+0000 7f309ff4e640 1 Processor -- start 2024-05-09T05:01:01.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.036+0000 7f309ff4e640 1 -- start start 2024-05-09T05:01:01.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.036+0000 7f309ff4e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f309815f300 0x7f30981639f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.036+0000 7f309ff4e640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f309806f9f0 0x7f30980d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309ff4e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f309806ff60 con 0x7f30980d3930 2024-05-09T05:01:01.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309ff4e640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f3098070240 con 0x7f309806f9f0 2024-05-09T05:01:01.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309ff4e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f3098070520 con 0x7f309815f300 2024-05-09T05:01:01.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309e4eb640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f30980d3930 0x7f309806ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51492/0 (socket says 172.21.15.60:51492) 2024-05-09T05:01:01.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309e4eb640 1 -- 172.21.15.60:0/3913113691 learned_addr learned my addr 172.21.15.60:0/3913113691 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:01.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309cce8640 1 -- 172.21.15.60:0/3913113691 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1276113522 0 0) 0x7f309806ff60 con 0x7f30980d3930 2024-05-09T05:01:01.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309d4e9640 1 --2- 172.21.15.60:0/3913113691 >> v2:172.21.15.114:3300/0 conn(0x7f309806f9f0 0x7f30980d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309cce8640 1 -- 172.21.15.60:0/3913113691 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3084003520 con 0x7f30980d3930 2024-05-09T05:01:01.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309dcea640 1 --2- 172.21.15.60:0/3913113691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f309815f300 0x7f30981639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309cce8640 1 -- 172.21.15.60:0/3913113691 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3465436255 0 0) 0x7f3084003520 con 0x7f30980d3930 2024-05-09T05:01:01.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309cce8640 1 -- 172.21.15.60:0/3913113691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f309815f300 msgr2=0x7f30981639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:01.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309cce8640 1 --2- 172.21.15.60:0/3913113691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f309815f300 0x7f30981639f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309cce8640 1 -- 172.21.15.60:0/3913113691 >> v2:172.21.15.114:3300/0 conn(0x7f309806f9f0 msgr2=0x7f30980d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:01.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309cce8640 1 --2- 172.21.15.60:0/3913113691 >> v2:172.21.15.114:3300/0 conn(0x7f309806f9f0 0x7f30980d33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309cce8640 1 -- 172.21.15.60:0/3913113691 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f309806e740 con 0x7f30980d3930 2024-05-09T05:01:01.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309cce8640 1 -- 172.21.15.60:0/3913113691 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f3090001a70 con 0x7f30980d3930 2024-05-09T05:01:01.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309cce8640 1 -- 172.21.15.60:0/3913113691 >> v1:172.21.15.60:6789/0 conn(0x7f30980d3930 legacy=0x7f309806ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:01.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309cce8640 1 --2- 172.21.15.60:0/3913113691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3084003a00 0x7f3084003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309cce8640 1 --2- 172.21.15.60:0/3913113691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f309815f300 0x7f3084004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309cce8640 1 --2- 172.21.15.60:0/3913113691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3084004d40 0x7f30840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309cce8640 1 -- 172.21.15.60:0/3913113691 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f3098070240 con 0x7f3084003a00 2024-05-09T05:01:01.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309cce8640 1 -- 172.21.15.60:0/3913113691 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3098070520 con 0x7f3084004d40 2024-05-09T05:01:01.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309cce8640 1 -- 172.21.15.60:0/3913113691 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f308400b9c0 con 0x7f309815f300 2024-05-09T05:01:01.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.037+0000 7f309dcea640 1 --2- 172.21.15.60:0/3913113691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3084003a00 0x7f3084003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.038+0000 7f309d4e9640 1 --2- 172.21.15.60:0/3913113691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f309815f300 0x7f3084004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.038+0000 7f309e4eb640 1 --2- 172.21.15.60:0/3913113691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3084004d40 0x7f30840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.038+0000 7f309dcea640 1 -- 172.21.15.60:0/3913113691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f309815f300 msgr2=0x7f3084004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:01.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.038+0000 7f309dcea640 1 --2- 172.21.15.60:0/3913113691 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f309815f300 0x7f3084004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.038+0000 7f309dcea640 1 -- 172.21.15.60:0/3913113691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3084004d40 msgr2=0x7f30840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:01.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.038+0000 7f309dcea640 1 --2- 172.21.15.60:0/3913113691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3084004d40 0x7f30840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.038+0000 7f309dcea640 1 -- 172.21.15.60:0/3913113691 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3084003520 con 0x7f3084003a00 2024-05-09T05:01:01.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.038+0000 7f309dcea640 1 --2- 172.21.15.60:0/3913113691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3084003a00 0x7f3084003e20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3094009ac0 tx=0x7f3084005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdee9b7e808e84ac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:01.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.039+0000 7f309cce8640 1 -- 172.21.15.60:0/3913113691 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3094036070 con 0x7f3084003a00 2024-05-09T05:01:01.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.039+0000 7f309cce8640 1 -- 172.21.15.60:0/3913113691 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3094031400 con 0x7f3084003a00 2024-05-09T05:01:01.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.039+0000 7f309cce8640 1 -- 172.21.15.60:0/3913113691 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3094009690 con 0x7f3084003a00 2024-05-09T05:01:01.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.039+0000 7f309ff4e640 1 -- 172.21.15.60:0/3913113691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3084003a00 msgr2=0x7f3084003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:01.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.039+0000 7f309ff4e640 1 --2- 172.21.15.60:0/3913113691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3084003a00 0x7f3084003e20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3094009ac0 tx=0x7f3084005830 comp rx=0 tx=0).stop 2024-05-09T05:01:01.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.039+0000 7f309e4eb640 1 -- 172.21.15.60:0/3913113691 reap_dead start 2024-05-09T05:01:01.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.039+0000 7f309ff4e640 1 -- 172.21.15.60:0/3913113691 shutdown_connections 2024-05-09T05:01:01.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.039+0000 7f309ff4e640 1 -- 172.21.15.60:0/3913113691 >> 172.21.15.60:0/3913113691 conn(0x7f309809b850 msgr2=0x7f309809bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:01.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.039+0000 7f309ff4e640 1 -- 172.21.15.60:0/3913113691 shutdown_connections 2024-05-09T05:01:01.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.039+0000 7f309ff4e640 1 -- 172.21.15.60:0/3913113691 wait complete. 2024-05-09T05:01:01.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.040+0000 7f309ff4e640 1 Processor -- start 2024-05-09T05:01:01.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.041+0000 7f309ff4e640 1 -- start start 2024-05-09T05:01:01.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.041+0000 7f309ff4e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3084003a00 0x7f30980b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.041+0000 7f309ff4e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f309809cec0 0x7f30980ad300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.041+0000 7f309ff4e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30980ad840 0x7f30981eaa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.041+0000 7f309ff4e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f309806f7d0 con 0x7f309809cec0 2024-05-09T05:01:01.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.041+0000 7f309ff4e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f309806f650 con 0x7f30980ad840 2024-05-09T05:01:01.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.041+0000 7f309ff4e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f30981606f0 con 0x7f3084003a00 2024-05-09T05:01:01.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.041+0000 7f309d4e9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f309809cec0 0x7f30980ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.041+0000 7f309d4e9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f309809cec0 0x7f30980ad300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56124/0 (socket says 172.21.15.60:56124) 2024-05-09T05:01:01.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.041+0000 7f309d4e9640 1 -- 172.21.15.60:0/2018142558 learned_addr learned my addr 172.21.15.60:0/2018142558 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:01.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.042+0000 7f309dcea640 1 --2- 172.21.15.60:0/2018142558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3084003a00 0x7f30980b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.042+0000 7f309e4eb640 1 --2- 172.21.15.60:0/2018142558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30980ad840 0x7f30981eaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.042+0000 7f309d4e9640 1 -- 172.21.15.60:0/2018142558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3084003a00 msgr2=0x7f30980b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:01.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.042+0000 7f309d4e9640 1 --2- 172.21.15.60:0/2018142558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3084003a00 0x7f30980b3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.042+0000 7f309d4e9640 1 -- 172.21.15.60:0/2018142558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30980ad840 msgr2=0x7f30981eaa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:01.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.042+0000 7f309d4e9640 1 --2- 172.21.15.60:0/2018142558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30980ad840 0x7f30981eaa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.042+0000 7f309d4e9640 1 -- 172.21.15.60:0/2018142558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30981e9e80 con 0x7f309809cec0 2024-05-09T05:01:01.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.042+0000 7f309d4e9640 1 --2- 172.21.15.60:0/2018142558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f309809cec0 0x7f30980ad300 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3088011a70 tx=0x7f3088011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:01.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.042+0000 7f308e7fc640 1 -- 172.21.15.60:0/2018142558 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f308800fa90 con 0x7f309809cec0 2024-05-09T05:01:01.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.042+0000 7f308e7fc640 1 -- 172.21.15.60:0/2018142558 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f308800fc30 con 0x7f309809cec0 2024-05-09T05:01:01.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.042+0000 7f309ff4e640 1 -- 172.21.15.60:0/2018142558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30980b3fe0 con 0x7f309809cec0 2024-05-09T05:01:01.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.042+0000 7f308e7fc640 1 -- 172.21.15.60:0/2018142558 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3088014440 con 0x7f309809cec0 2024-05-09T05:01:01.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.042+0000 7f309ff4e640 1 -- 172.21.15.60:0/2018142558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f309814aae0 con 0x7f309809cec0 2024-05-09T05:01:01.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.043+0000 7f309ff4e640 1 -- 172.21.15.60:0/2018142558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3098051040 con 0x7f309809cec0 2024-05-09T05:01:01.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.044+0000 7f308e7fc640 1 -- 172.21.15.60:0/2018142558 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3088012040 con 0x7f309809cec0 2024-05-09T05:01:01.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.044+0000 7f308e7fc640 1 --2- 172.21.15.60:0/2018142558 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f307c0762d0 0x7f307c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.044+0000 7f308e7fc640 1 -- 172.21.15.60:0/2018142558 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5249+0+0 (secure 0 0 0) 0x7f30880978c0 con 0x7f309809cec0 2024-05-09T05:01:01.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.044+0000 7f309dcea640 1 --2- 172.21.15.60:0/2018142558 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f307c0762d0 0x7f307c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.045+0000 7f309dcea640 1 --2- 172.21.15.60:0/2018142558 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f307c0762d0 0x7f307c078790 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3094033000 tx=0x7f3094008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:01.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.049+0000 7f308e7fc640 1 -- 172.21.15.60:0/2018142558 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3088019030 con 0x7f309809cec0 2024-05-09T05:01:01.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.312+0000 7f309ff4e640 1 -- 172.21.15.60:0/2018142558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f30980adc20 con 0x7f309809cec0 2024-05-09T05:01:01.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.314+0000 7f308e7fc640 1 -- 172.21.15.60:0/2018142558 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f3088061470 con 0x7f309809cec0 2024-05-09T05:01:01.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.316+0000 7f309ff4e640 1 -- 172.21.15.60:0/2018142558 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f307c0762d0 msgr2=0x7f307c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:01.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.316+0000 7f309ff4e640 1 --2- 172.21.15.60:0/2018142558 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f307c0762d0 0x7f307c078790 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3094033000 tx=0x7f3094008030 comp rx=0 tx=0).stop 2024-05-09T05:01:01.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.316+0000 7f309ff4e640 1 -- 172.21.15.60:0/2018142558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f309809cec0 msgr2=0x7f30980ad300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:01.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.316+0000 7f309ff4e640 1 --2- 172.21.15.60:0/2018142558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f309809cec0 0x7f30980ad300 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3088011a70 tx=0x7f3088011e70 comp rx=0 tx=0).stop 2024-05-09T05:01:01.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.316+0000 7f309ff4e640 1 -- 172.21.15.60:0/2018142558 shutdown_connections 2024-05-09T05:01:01.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.316+0000 7f309ff4e640 1 --2- 172.21.15.60:0/2018142558 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f307c0762d0 0x7f307c078790 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.316+0000 7f309ff4e640 1 --2- 172.21.15.60:0/2018142558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30980ad840 0x7f30981eaa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.316+0000 7f309ff4e640 1 --2- 172.21.15.60:0/2018142558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f309809cec0 0x7f30980ad300 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.316+0000 7f309ff4e640 1 --2- 172.21.15.60:0/2018142558 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3084003a00 0x7f30980b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.316+0000 7f309ff4e640 1 -- 172.21.15.60:0/2018142558 >> 172.21.15.60:0/2018142558 conn(0x7f309809b850 msgr2=0x7f309809a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.316+0000 7f309ff4e640 1 -- 172.21.15.60:0/2018142558 shutdown_connections 2024-05-09T05:01:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.316+0000 7f309ff4e640 1 -- 172.21.15.60:0/2018142558 wait complete. 2024-05-09T05:01:01.328 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-05-09T05:01:01.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.432+0000 7f46d72a7640 1 Processor -- start 2024-05-09T05:01:01.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.433+0000 7f46d72a7640 1 -- start start 2024-05-09T05:01:01.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.433+0000 7f46d72a7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f46d015f3e0 0x7f46d00b4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.433+0000 7f46d72a7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d009c8b0 0x7f46d00b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.433+0000 7f46d72a7640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46d0071f30 con 0x7f46d0073310 2024-05-09T05:01:01.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.433+0000 7f46d72a7640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f46d0071ce0 con 0x7f46d015f3e0 2024-05-09T05:01:01.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.433+0000 7f46d72a7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f46d0072210 con 0x7f46d009c8b0 2024-05-09T05:01:01.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.434+0000 7f46d5043640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f46d015f3e0 0x7f46d00b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.434+0000 7f46d5043640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f46d015f3e0 0x7f46d00b4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56132/0 (socket says 172.21.15.60:56132) 2024-05-09T05:01:01.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.434+0000 7f46d5043640 1 -- 172.21.15.60:0/4279948639 learned_addr learned my addr 172.21.15.60:0/4279948639 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:01.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.434+0000 7f46d4842640 1 --2- 172.21.15.60:0/4279948639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d009c8b0 0x7f46d00b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.434+0000 7f46d5043640 1 -- 172.21.15.60:0/4279948639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d009c8b0 msgr2=0x7f46d00b4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:01.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.434+0000 7f46d5043640 1 --2- 172.21.15.60:0/4279948639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d009c8b0 0x7f46d00b4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.434+0000 7f46d5043640 1 -- 172.21.15.60:0/4279948639 >> v1:172.21.15.114:6789/0 conn(0x7f46d0073310 legacy=0x7f46d006e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:01.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.434+0000 7f46d5043640 1 -- 172.21.15.60:0/4279948639 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46d00724f0 con 0x7f46d015f3e0 2024-05-09T05:01:01.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.435+0000 7f46d5043640 1 --2- 172.21.15.60:0/4279948639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46d015f3e0 0x7f46d00b4630 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f46cc009870 tx=0x7f46cc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67f5458a8fb593d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:01.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.435+0000 7f46c7fff640 1 -- 172.21.15.60:0/4279948639 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46cc037070 con 0x7f46d015f3e0 2024-05-09T05:01:01.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.435+0000 7f46c7fff640 1 -- 172.21.15.60:0/4279948639 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46cc032440 con 0x7f46d015f3e0 2024-05-09T05:01:01.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.435+0000 7f46c7fff640 1 -- 172.21.15.60:0/4279948639 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46cc032810 con 0x7f46d015f3e0 2024-05-09T05:01:01.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.435+0000 7f46d72a7640 1 -- 172.21.15.60:0/4279948639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46d015f3e0 msgr2=0x7f46d00b4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:01.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.435+0000 7f46d72a7640 1 --2- 172.21.15.60:0/4279948639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46d015f3e0 0x7f46d00b4630 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f46cc009870 tx=0x7f46cc02a270 comp rx=0 tx=0).stop 2024-05-09T05:01:01.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.435+0000 7f46d72a7640 1 -- 172.21.15.60:0/4279948639 shutdown_connections 2024-05-09T05:01:01.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.435+0000 7f46d72a7640 1 --2- 172.21.15.60:0/4279948639 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d009c8b0 0x7f46d00b4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.435+0000 7f46d72a7640 1 --2- 172.21.15.60:0/4279948639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46d015f3e0 0x7f46d00b4630 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.435+0000 7f46d72a7640 1 -- 172.21.15.60:0/4279948639 >> 172.21.15.60:0/4279948639 conn(0x7f46d00d1470 msgr2=0x7f46d00d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:01.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.435+0000 7f46d72a7640 1 -- 172.21.15.60:0/4279948639 shutdown_connections 2024-05-09T05:01:01.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.436+0000 7f46d72a7640 1 -- 172.21.15.60:0/4279948639 wait complete. 2024-05-09T05:01:01.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.437+0000 7f46d72a7640 1 Processor -- start 2024-05-09T05:01:01.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.437+0000 7f46d72a7640 1 -- start start 2024-05-09T05:01:01.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.437+0000 7f46d72a7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d0073310 0x7f46d00d3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.437+0000 7f46d72a7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46d009c8b0 0x7f46d00b0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.437+0000 7f46d72a7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d015f3e0 0x7f46d00b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.437+0000 7f46d72a7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f46d015f7c0 con 0x7f46d009c8b0 2024-05-09T05:01:01.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.437+0000 7f46d72a7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f46d0072ab0 con 0x7f46d015f3e0 2024-05-09T05:01:01.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.437+0000 7f46d72a7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f46d009f110 con 0x7f46d0073310 2024-05-09T05:01:01.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.437+0000 7f46d4842640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46d009c8b0 0x7f46d00b0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.438+0000 7f46d4842640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46d009c8b0 0x7f46d00b0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56144/0 (socket says 172.21.15.60:56144) 2024-05-09T05:01:01.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.438+0000 7f46d4842640 1 -- 172.21.15.60:0/2277517443 learned_addr learned my addr 172.21.15.60:0/2277517443 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:01.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.438+0000 7f46d5043640 1 --2- 172.21.15.60:0/2277517443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d0073310 0x7f46d00d3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.438+0000 7f46d5844640 1 --2- 172.21.15.60:0/2277517443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d015f3e0 0x7f46d00b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.438+0000 7f46d4842640 1 -- 172.21.15.60:0/2277517443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d0073310 msgr2=0x7f46d00d3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:01.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.438+0000 7f46d4842640 1 --2- 172.21.15.60:0/2277517443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d0073310 0x7f46d00d3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.438+0000 7f46d4842640 1 -- 172.21.15.60:0/2277517443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d015f3e0 msgr2=0x7f46d00b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:01.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.438+0000 7f46d4842640 1 --2- 172.21.15.60:0/2277517443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d015f3e0 0x7f46d00b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.438+0000 7f46d4842640 1 -- 172.21.15.60:0/2277517443 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46d014fb60 con 0x7f46d009c8b0 2024-05-09T05:01:01.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.438+0000 7f46d4842640 1 --2- 172.21.15.60:0/2277517443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46d009c8b0 0x7f46d00b0360 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f46c000d800 tx=0x7f46c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:01.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.438+0000 7f46c5ffb640 1 -- 172.21.15.60:0/2277517443 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c000b660 con 0x7f46d009c8b0 2024-05-09T05:01:01.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.438+0000 7f46c5ffb640 1 -- 172.21.15.60:0/2277517443 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46c0019070 con 0x7f46d009c8b0 2024-05-09T05:01:01.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.438+0000 7f46d72a7640 1 -- 172.21.15.60:0/2277517443 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46d0160630 con 0x7f46d009c8b0 2024-05-09T05:01:01.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.438+0000 7f46c5ffb640 1 -- 172.21.15.60:0/2277517443 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c000ba60 con 0x7f46d009c8b0 2024-05-09T05:01:01.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.438+0000 7f46d72a7640 1 -- 172.21.15.60:0/2277517443 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46d0150100 con 0x7f46d009c8b0 2024-05-09T05:01:01.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.439+0000 7f46d72a7640 1 -- 172.21.15.60:0/2277517443 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46d0150300 con 0x7f46d009c8b0 2024-05-09T05:01:01.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.440+0000 7f46c5ffb640 1 -- 172.21.15.60:0/2277517443 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f46c000bc00 con 0x7f46d009c8b0 2024-05-09T05:01:01.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.440+0000 7f46c5ffb640 1 --2- 172.21.15.60:0/2277517443 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f46b40762d0 0x7f46b4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.440+0000 7f46c5ffb640 1 -- 172.21.15.60:0/2277517443 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5249+0+0 (secure 0 0 0) 0x7f46c0097a10 con 0x7f46d009c8b0 2024-05-09T05:01:01.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.441+0000 7f46d5043640 1 --2- 172.21.15.60:0/2277517443 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f46b40762d0 0x7f46b4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.441+0000 7f46d5043640 1 --2- 172.21.15.60:0/2277517443 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f46b40762d0 0x7f46b4078790 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f46cc034000 tx=0x7f46cc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:01.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.445+0000 7f46c5ffb640 1 -- 172.21.15.60:0/2277517443 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f46d0150300 con 0x7f46d009c8b0 2024-05-09T05:01:01.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.710+0000 7f46d72a7640 1 -- 172.21.15.60:0/2277517443 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f46d0051040 con 0x7f46d009c8b0 2024-05-09T05:01:01.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.712+0000 7f46c5ffb640 1 -- 172.21.15.60:0/2277517443 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) ==== 97+0+0 (secure 0 0 0) 0x7f46c0061400 con 0x7f46d009c8b0 2024-05-09T05:01:01.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.714+0000 7f46d72a7640 1 -- 172.21.15.60:0/2277517443 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f46b40762d0 msgr2=0x7f46b4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:01.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.714+0000 7f46d72a7640 1 --2- 172.21.15.60:0/2277517443 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f46b40762d0 0x7f46b4078790 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f46cc034000 tx=0x7f46cc02e000 comp rx=0 tx=0).stop 2024-05-09T05:01:01.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.714+0000 7f46d72a7640 1 -- 172.21.15.60:0/2277517443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46d009c8b0 msgr2=0x7f46d00b0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:01.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.714+0000 7f46d72a7640 1 --2- 172.21.15.60:0/2277517443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46d009c8b0 0x7f46d00b0360 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f46c000d800 tx=0x7f46c000dcc0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.714+0000 7f46d72a7640 1 -- 172.21.15.60:0/2277517443 shutdown_connections 2024-05-09T05:01:01.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.714+0000 7f46d72a7640 1 --2- 172.21.15.60:0/2277517443 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f46b40762d0 0x7f46b4078790 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.714+0000 7f46d72a7640 1 --2- 172.21.15.60:0/2277517443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f46d015f3e0 0x7f46d00b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.714+0000 7f46d72a7640 1 --2- 172.21.15.60:0/2277517443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f46d009c8b0 0x7f46d00b0360 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.714+0000 7f46d72a7640 1 --2- 172.21.15.60:0/2277517443 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f46d0073310 0x7f46d00d3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.714+0000 7f46d72a7640 1 -- 172.21.15.60:0/2277517443 >> 172.21.15.60:0/2277517443 conn(0x7f46d00d1470 msgr2=0x7f46d0068520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:01.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.714+0000 7f46d72a7640 1 -- 172.21.15.60:0/2277517443 shutdown_connections 2024-05-09T05:01:01.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.714+0000 7f46d72a7640 1 -- 172.21.15.60:0/2277517443 wait complete. 2024-05-09T05:01:01.725 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-05-09T05:01:01.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.831+0000 7fa97b77f640 1 Processor -- start 2024-05-09T05:01:01.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.831+0000 7fa97b77f640 1 -- start start 2024-05-09T05:01:01.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.832+0000 7fa97b77f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa9740b4d40 0x7fa97409e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.832+0000 7fa97b77f640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa97409c8b0 0x7fa97409e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.832+0000 7fa97b77f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa974071f30 con 0x7fa97415f3e0 2024-05-09T05:01:01.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.832+0000 7fa97b77f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fa974071ce0 con 0x7fa9740b4d40 2024-05-09T05:01:01.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.832+0000 7fa97b77f640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fa974072210 con 0x7fa97409c8b0 2024-05-09T05:01:01.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.832+0000 7fa978d1a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa9740b4d40 0x7fa97409e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.832+0000 7fa97951b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa97415f3e0 0x7fa9740b4630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51498/0 (socket says 172.21.15.60:51498) 2024-05-09T05:01:01.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.832+0000 7fa97951b640 1 -- 172.21.15.60:0/2473214253 learned_addr learned my addr 172.21.15.60:0/2473214253 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:01.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.832+0000 7fa979d1c640 1 --2- 172.21.15.60:0/2473214253 >> v2:172.21.15.114:3300/0 conn(0x7fa97409c8b0 0x7fa97409e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.832+0000 7fa978d1a640 1 -- 172.21.15.60:0/2473214253 >> v2:172.21.15.114:3300/0 conn(0x7fa97409c8b0 msgr2=0x7fa97409e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:01.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.832+0000 7fa978d1a640 1 --2- 172.21.15.60:0/2473214253 >> v2:172.21.15.114:3300/0 conn(0x7fa97409c8b0 0x7fa97409e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.832+0000 7fa978d1a640 1 -- 172.21.15.60:0/2473214253 >> v1:172.21.15.60:6789/0 conn(0x7fa97415f3e0 legacy=0x7fa9740b4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:01.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.832+0000 7fa96bfff640 1 -- 172.21.15.60:0/2473214253 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2370874374 0 0) 0x7fa974071f30 con 0x7fa97415f3e0 2024-05-09T05:01:01.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.832+0000 7fa978d1a640 1 -- 172.21.15.60:0/2473214253 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9740724f0 con 0x7fa9740b4d40 2024-05-09T05:01:01.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.833+0000 7fa978d1a640 1 --2- 172.21.15.60:0/2473214253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9740b4d40 0x7fa97409e100 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa97409cde0 tx=0x7fa96402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc2f0a1697c5cfef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:01.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.833+0000 7fa96bfff640 1 -- 172.21.15.60:0/2473214253 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa964037070 con 0x7fa9740b4d40 2024-05-09T05:01:01.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.833+0000 7fa96bfff640 1 -- 172.21.15.60:0/2473214253 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa964032440 con 0x7fa9740b4d40 2024-05-09T05:01:01.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.833+0000 7fa96bfff640 1 -- 172.21.15.60:0/2473214253 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa964032810 con 0x7fa9740b4d40 2024-05-09T05:01:01.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.834+0000 7fa97b77f640 1 -- 172.21.15.60:0/2473214253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9740b4d40 msgr2=0x7fa97409e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:01.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.834+0000 7fa97b77f640 1 --2- 172.21.15.60:0/2473214253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9740b4d40 0x7fa97409e100 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa97409cde0 tx=0x7fa96402a1a0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.834+0000 7fa97b77f640 1 -- 172.21.15.60:0/2473214253 shutdown_connections 2024-05-09T05:01:01.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.834+0000 7fa97b77f640 1 --2- 172.21.15.60:0/2473214253 >> v2:172.21.15.114:3300/0 conn(0x7fa97409c8b0 0x7fa97409e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.834+0000 7fa97b77f640 1 --2- 172.21.15.60:0/2473214253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9740b4d40 0x7fa97409e100 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.834+0000 7fa97b77f640 1 -- 172.21.15.60:0/2473214253 >> 172.21.15.60:0/2473214253 conn(0x7fa9740d1470 msgr2=0x7fa9740d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:01.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.834+0000 7fa97b77f640 1 -- 172.21.15.60:0/2473214253 shutdown_connections 2024-05-09T05:01:01.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.834+0000 7fa97b77f640 1 -- 172.21.15.60:0/2473214253 wait complete. 2024-05-09T05:01:01.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.835+0000 7fa97b77f640 1 Processor -- start 2024-05-09T05:01:01.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.835+0000 7fa97b77f640 1 -- start start 2024-05-09T05:01:01.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.836+0000 7fa97b77f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa97409c8b0 0x7fa97415d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.836+0000 7fa97b77f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9740b4d40 0x7fa9741f4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.836+0000 7fa97b77f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa97415f3e0 0x7fa9740a5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.836+0000 7fa97b77f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa97415f7c0 con 0x7fa97409c8b0 2024-05-09T05:01:01.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.836+0000 7fa97b77f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa974072ab0 con 0x7fa97415f3e0 2024-05-09T05:01:01.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.836+0000 7fa97b77f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa974072fd0 con 0x7fa9740b4d40 2024-05-09T05:01:01.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.836+0000 7fa97951b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa97409c8b0 0x7fa97415d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.836+0000 7fa97951b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa97409c8b0 0x7fa97415d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56162/0 (socket says 172.21.15.60:56162) 2024-05-09T05:01:01.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.836+0000 7fa97951b640 1 -- 172.21.15.60:0/2113559380 learned_addr learned my addr 172.21.15.60:0/2113559380 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:01.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.836+0000 7fa979d1c640 1 --2- 172.21.15.60:0/2113559380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa97415f3e0 0x7fa9740a5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.836+0000 7fa978d1a640 1 --2- 172.21.15.60:0/2113559380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9740b4d40 0x7fa9741f4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.836+0000 7fa97951b640 1 -- 172.21.15.60:0/2113559380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9740b4d40 msgr2=0x7fa9741f4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:01.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.836+0000 7fa97951b640 1 --2- 172.21.15.60:0/2113559380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9740b4d40 0x7fa9741f4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.836+0000 7fa97951b640 1 -- 172.21.15.60:0/2113559380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa97415f3e0 msgr2=0x7fa9740a5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:01.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.836+0000 7fa97951b640 1 --2- 172.21.15.60:0/2113559380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa97415f3e0 0x7fa9740a5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:01.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.836+0000 7fa97951b640 1 -- 172.21.15.60:0/2113559380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa97415d7e0 con 0x7fa97409c8b0 2024-05-09T05:01:01.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.836+0000 7fa979d1c640 1 --2- 172.21.15.60:0/2113559380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa97415f3e0 0x7fa9740a5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:01.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.836+0000 7fa97951b640 1 --2- 172.21.15.60:0/2113559380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa97409c8b0 0x7fa97415d2a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa97000c3e0 tx=0x7fa97000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:01.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.837+0000 7fa969ffb640 1 -- 172.21.15.60:0/2113559380 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9700074e0 con 0x7fa97409c8b0 2024-05-09T05:01:01.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.837+0000 7fa969ffb640 1 -- 172.21.15.60:0/2113559380 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9700047e0 con 0x7fa97409c8b0 2024-05-09T05:01:01.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.837+0000 7fa97b77f640 1 -- 172.21.15.60:0/2113559380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9740a6160 con 0x7fa97409c8b0 2024-05-09T05:01:01.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.837+0000 7fa969ffb640 1 -- 172.21.15.60:0/2113559380 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa970004bd0 con 0x7fa97409c8b0 2024-05-09T05:01:01.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.837+0000 7fa97b77f640 1 -- 172.21.15.60:0/2113559380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9740a6720 con 0x7fa97409c8b0 2024-05-09T05:01:01.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.838+0000 7fa97b77f640 1 -- 172.21.15.60:0/2113559380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa974051040 con 0x7fa97409c8b0 2024-05-09T05:01:01.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.838+0000 7fa969ffb640 1 -- 172.21.15.60:0/2113559380 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa970011070 con 0x7fa97409c8b0 2024-05-09T05:01:01.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.839+0000 7fa969ffb640 1 --2- 172.21.15.60:0/2113559380 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa9580762d0 0x7fa958078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:01.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.839+0000 7fa969ffb640 1 -- 172.21.15.60:0/2113559380 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5249+0+0 (secure 0 0 0) 0x7fa9700970d0 con 0x7fa97409c8b0 2024-05-09T05:01:01.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.839+0000 7fa978d1a640 1 --2- 172.21.15.60:0/2113559380 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa9580762d0 0x7fa958078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:01.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.840+0000 7fa978d1a640 1 --2- 172.21.15.60:0/2113559380 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa9580762d0 0x7fa958078790 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa964034000 tx=0x7fa96402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:01.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:01.843+0000 7fa969ffb640 1 -- 172.21.15.60:0/2113559380 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa970060b60 con 0x7fa97409c8b0 2024-05-09T05:01:02.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.112+0000 7fa97b77f640 1 -- 172.21.15.60:0/2113559380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7fa9740a6e20 con 0x7fa97409c8b0 2024-05-09T05:01:02.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.114+0000 7fa969ffb640 1 -- 172.21.15.60:0/2113559380 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) ==== 90+0+0 (secure 0 0 0) 0x7fa970060460 con 0x7fa97409c8b0 2024-05-09T05:01:02.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.116+0000 7fa97b77f640 1 -- 172.21.15.60:0/2113559380 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa9580762d0 msgr2=0x7fa958078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:02.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.116+0000 7fa97b77f640 1 --2- 172.21.15.60:0/2113559380 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa9580762d0 0x7fa958078790 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa964034000 tx=0x7fa96402e000 comp rx=0 tx=0).stop 2024-05-09T05:01:02.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.116+0000 7fa97b77f640 1 -- 172.21.15.60:0/2113559380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa97409c8b0 msgr2=0x7fa97415d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:02.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.116+0000 7fa97b77f640 1 --2- 172.21.15.60:0/2113559380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa97409c8b0 0x7fa97415d2a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa97000c3e0 tx=0x7fa97000c8a0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.116+0000 7fa97b77f640 1 -- 172.21.15.60:0/2113559380 shutdown_connections 2024-05-09T05:01:02.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.116+0000 7fa97b77f640 1 --2- 172.21.15.60:0/2113559380 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa9580762d0 0x7fa958078790 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.116+0000 7fa97b77f640 1 --2- 172.21.15.60:0/2113559380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa97415f3e0 0x7fa9740a5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.116+0000 7fa97b77f640 1 --2- 172.21.15.60:0/2113559380 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9740b4d40 0x7fa9741f4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.116+0000 7fa97b77f640 1 --2- 172.21.15.60:0/2113559380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa97409c8b0 0x7fa97415d2a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.116+0000 7fa97b77f640 1 -- 172.21.15.60:0/2113559380 >> 172.21.15.60:0/2113559380 conn(0x7fa9740d1470 msgr2=0x7fa97409ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:02.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.117+0000 7fa97b77f640 1 -- 172.21.15.60:0/2113559380 shutdown_connections 2024-05-09T05:01:02.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.117+0000 7fa97b77f640 1 -- 172.21.15.60:0/2113559380 wait complete. 2024-05-09T05:01:02.128 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-09T05:01:02.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.231+0000 7f236d2fc640 1 Processor -- start 2024-05-09T05:01:02.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.231+0000 7f236d2fc640 1 -- start start 2024-05-09T05:01:02.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.232+0000 7f236d2fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f236815f3e0 0x7f23680b4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:02.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.232+0000 7f236d2fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f236809c8b0 0x7f23680b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:02.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.232+0000 7f236d2fc640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2368071f30 con 0x7f2368073310 2024-05-09T05:01:02.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.232+0000 7f236d2fc640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f2368071ce0 con 0x7f236809c8b0 2024-05-09T05:01:02.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.232+0000 7f236d2fc640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f2368072210 con 0x7f236815f3e0 2024-05-09T05:01:02.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.232+0000 7f23667fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f236809c8b0 0x7f23680b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:02.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.232+0000 7f23667fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f236809c8b0 0x7f23680b4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56166/0 (socket says 172.21.15.60:56166) 2024-05-09T05:01:02.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.232+0000 7f23667fc640 1 -- 172.21.15.60:0/4225033450 learned_addr learned my addr 172.21.15.60:0/4225033450 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:02.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.232+0000 7f2366ffd640 1 --2- 172.21.15.60:0/4225033450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f236815f3e0 0x7f23680b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:02.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.232+0000 7f23667fc640 1 -- 172.21.15.60:0/4225033450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f236815f3e0 msgr2=0x7f23680b4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:02.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.232+0000 7f23667fc640 1 --2- 172.21.15.60:0/4225033450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f236815f3e0 0x7f23680b4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.232+0000 7f23667fc640 1 -- 172.21.15.60:0/4225033450 >> v1:172.21.15.114:6789/0 conn(0x7f2368073310 legacy=0x7f236806e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:02.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.233+0000 7f23667fc640 1 -- 172.21.15.60:0/4225033450 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23680724f0 con 0x7f236809c8b0 2024-05-09T05:01:02.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.233+0000 7f23667fc640 1 --2- 172.21.15.60:0/4225033450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f236809c8b0 0x7f23680b4b70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2350009870 tx=0x7f235002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c106015ade710a24 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:02.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.233+0000 7f2365ffb640 1 -- 172.21.15.60:0/4225033450 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2350037070 con 0x7f236809c8b0 2024-05-09T05:01:02.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.233+0000 7f2365ffb640 1 -- 172.21.15.60:0/4225033450 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2350032440 con 0x7f236809c8b0 2024-05-09T05:01:02.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.233+0000 7f2365ffb640 1 -- 172.21.15.60:0/4225033450 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2350032810 con 0x7f236809c8b0 2024-05-09T05:01:02.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.234+0000 7f236d2fc640 1 -- 172.21.15.60:0/4225033450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f236809c8b0 msgr2=0x7f23680b4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:02.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.234+0000 7f236d2fc640 1 --2- 172.21.15.60:0/4225033450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f236809c8b0 0x7f23680b4b70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2350009870 tx=0x7f235002a270 comp rx=0 tx=0).stop 2024-05-09T05:01:02.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.234+0000 7f236d2fc640 1 -- 172.21.15.60:0/4225033450 shutdown_connections 2024-05-09T05:01:02.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.234+0000 7f236d2fc640 1 --2- 172.21.15.60:0/4225033450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f236809c8b0 0x7f23680b4b70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.234+0000 7f236d2fc640 1 --2- 172.21.15.60:0/4225033450 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f236815f3e0 0x7f23680b4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.234+0000 7f236d2fc640 1 -- 172.21.15.60:0/4225033450 >> 172.21.15.60:0/4225033450 conn(0x7f23680d1470 msgr2=0x7f23680d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:02.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.234+0000 7f236d2fc640 1 -- 172.21.15.60:0/4225033450 shutdown_connections 2024-05-09T05:01:02.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.234+0000 7f236d2fc640 1 -- 172.21.15.60:0/4225033450 wait complete. 2024-05-09T05:01:02.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.235+0000 7f236d2fc640 1 Processor -- start 2024-05-09T05:01:02.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.235+0000 7f236d2fc640 1 -- start start 2024-05-09T05:01:02.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.236+0000 7f236d2fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2368073310 0x7f236815d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:02.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.236+0000 7f236d2fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f236809c8b0 0x7f2368162290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:02.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.236+0000 7f236d2fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f236815f3e0 0x7f2368153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:02.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.236+0000 7f236d2fc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f236815f7c0 con 0x7f236809c8b0 2024-05-09T05:01:02.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.236+0000 7f236d2fc640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2368072ab0 con 0x7f2368073310 2024-05-09T05:01:02.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.236+0000 7f236d2fc640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f236809f110 con 0x7f236815f3e0 2024-05-09T05:01:02.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.236+0000 7f23667fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f236809c8b0 0x7f2368162290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:02.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.236+0000 7f23667fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f236809c8b0 0x7f2368162290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56176/0 (socket says 172.21.15.60:56176) 2024-05-09T05:01:02.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.236+0000 7f23667fc640 1 -- 172.21.15.60:0/487908791 learned_addr learned my addr 172.21.15.60:0/487908791 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:02.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.236+0000 7f23677fe640 1 --2- 172.21.15.60:0/487908791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f236815f3e0 0x7f2368153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:02.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.236+0000 7f2366ffd640 1 --2- 172.21.15.60:0/487908791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2368073310 0x7f236815d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:02.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.237+0000 7f23667fc640 1 -- 172.21.15.60:0/487908791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f236815f3e0 msgr2=0x7f2368153e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:02.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.237+0000 7f23667fc640 1 --2- 172.21.15.60:0/487908791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f236815f3e0 0x7f2368153e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.237+0000 7f23667fc640 1 -- 172.21.15.60:0/487908791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2368073310 msgr2=0x7f236815d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:02.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.237+0000 7f23667fc640 1 --2- 172.21.15.60:0/487908791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2368073310 0x7f236815d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.237+0000 7f23667fc640 1 -- 172.21.15.60:0/487908791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23681611f0 con 0x7f236809c8b0 2024-05-09T05:01:02.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.237+0000 7f2366ffd640 1 --2- 172.21.15.60:0/487908791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2368073310 0x7f236815d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:02.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.237+0000 7f23667fc640 1 --2- 172.21.15.60:0/487908791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f236809c8b0 0x7f2368162290 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f235002a2b0 tx=0x7f23500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:02.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.237+0000 7f235f7fe640 1 -- 172.21.15.60:0/487908791 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2350041020 con 0x7f236809c8b0 2024-05-09T05:01:02.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.237+0000 7f235f7fe640 1 -- 172.21.15.60:0/487908791 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23500306c0 con 0x7f236809c8b0 2024-05-09T05:01:02.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.237+0000 7f236d2fc640 1 -- 172.21.15.60:0/487908791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23681614d0 con 0x7f236809c8b0 2024-05-09T05:01:02.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.237+0000 7f235f7fe640 1 -- 172.21.15.60:0/487908791 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2350037040 con 0x7f236809c8b0 2024-05-09T05:01:02.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.237+0000 7f236d2fc640 1 -- 172.21.15.60:0/487908791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2368161a90 con 0x7f236809c8b0 2024-05-09T05:01:02.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.238+0000 7f236d2fc640 1 -- 172.21.15.60:0/487908791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2368161f80 con 0x7f236809c8b0 2024-05-09T05:01:02.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.239+0000 7f235f7fe640 1 -- 172.21.15.60:0/487908791 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2350044020 con 0x7f236809c8b0 2024-05-09T05:01:02.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.239+0000 7f235f7fe640 1 --2- 172.21.15.60:0/487908791 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f23480762d0 0x7f2348078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:02.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.239+0000 7f235f7fe640 1 -- 172.21.15.60:0/487908791 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5249+0+0 (secure 0 0 0) 0x7f23500b7ad0 con 0x7f236809c8b0 2024-05-09T05:01:02.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.239+0000 7f2366ffd640 1 --2- 172.21.15.60:0/487908791 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f23480762d0 0x7f2348078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:02.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.240+0000 7f2366ffd640 1 --2- 172.21.15.60:0/487908791 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f23480762d0 0x7f2348078790 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2358006fd0 tx=0x7f2358008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:02.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.243+0000 7f235f7fe640 1 -- 172.21.15.60:0/487908791 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f235004b030 con 0x7f236809c8b0 2024-05-09T05:01:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.439+0000 7f236d2fc640 1 -- 172.21.15.60:0/487908791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f2368051040 con 0x7f236809c8b0 2024-05-09T05:01:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.440+0000 7f235f7fe640 1 -- 172.21.15.60:0/487908791 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v32) ==== 84+0+65 (secure 0 0 0) 0x7f2350081680 con 0x7f236809c8b0 2024-05-09T05:01:02.440 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:02.441 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2024-05-09T05:01:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2024-05-09T05:01:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2024-05-09T05:01:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:02.441 INFO:teuthology.orchestra.run.smithi060.stdout:] 2024-05-09T05:01:02.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.442+0000 7f236d2fc640 1 -- 172.21.15.60:0/487908791 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f23480762d0 msgr2=0x7f2348078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:02.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.442+0000 7f236d2fc640 1 --2- 172.21.15.60:0/487908791 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f23480762d0 0x7f2348078790 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2358006fd0 tx=0x7f2358008040 comp rx=0 tx=0).stop 2024-05-09T05:01:02.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.442+0000 7f236d2fc640 1 -- 172.21.15.60:0/487908791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f236809c8b0 msgr2=0x7f2368162290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:02.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.442+0000 7f236d2fc640 1 --2- 172.21.15.60:0/487908791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f236809c8b0 0x7f2368162290 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f235002a2b0 tx=0x7f23500046c0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.442+0000 7f236d2fc640 1 -- 172.21.15.60:0/487908791 shutdown_connections 2024-05-09T05:01:02.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.442+0000 7f236d2fc640 1 --2- 172.21.15.60:0/487908791 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f23480762d0 0x7f2348078790 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.442+0000 7f236d2fc640 1 --2- 172.21.15.60:0/487908791 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f236815f3e0 0x7f2368153e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.442+0000 7f236d2fc640 1 --2- 172.21.15.60:0/487908791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f236809c8b0 0x7f2368162290 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.442+0000 7f236d2fc640 1 --2- 172.21.15.60:0/487908791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2368073310 0x7f236815d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.442+0000 7f236d2fc640 1 -- 172.21.15.60:0/487908791 >> 172.21.15.60:0/487908791 conn(0x7f23680d1470 msgr2=0x7f23680b1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:02.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.442+0000 7f236d2fc640 1 -- 172.21.15.60:0/487908791 shutdown_connections 2024-05-09T05:01:02.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.442+0000 7f236d2fc640 1 -- 172.21.15.60:0/487908791 wait complete. 2024-05-09T05:01:02.453 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-09T05:01:02.454 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-09T05:01:02.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.561+0000 7f1ecbde4640 1 Processor -- start 2024-05-09T05:01:02.562 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.561+0000 7f1ecbde4640 1 -- start start 2024-05-09T05:01:02.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.562+0000 7f1ecbde4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1ec4148be0 0x7f1ec40cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:02.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.562+0000 7f1ecbde4640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1ec40d2880 0x7f1ec4162f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:02.563 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.562+0000 7f1ecbde4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ec40cecc0 con 0x7f1ec41634d0 2024-05-09T05:01:02.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.562+0000 7f1ecbde4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f1ec40d0d40 con 0x7f1ec4148be0 2024-05-09T05:01:02.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.562+0000 7f1ecbde4640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f1ec4163900 con 0x7f1ec40d2880 2024-05-09T05:01:02.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.562+0000 7f1ec9b80640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1ec4148be0 0x7f1ec40cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:02.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.563+0000 7f1ec9b80640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1ec4148be0 0x7f1ec40cc7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56186/0 (socket says 172.21.15.60:56186) 2024-05-09T05:01:02.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.563+0000 7f1ec9b80640 1 -- 172.21.15.60:0/4236564147 learned_addr learned my addr 172.21.15.60:0/4236564147 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:02.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.563+0000 7f1ec937f640 1 --2- 172.21.15.60:0/4236564147 >> v2:172.21.15.114:3300/0 conn(0x7f1ec40d2880 0x7f1ec4162f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:02.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.563+0000 7f1ec9b80640 1 -- 172.21.15.60:0/4236564147 >> v2:172.21.15.114:3300/0 conn(0x7f1ec40d2880 msgr2=0x7f1ec4162f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:02.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.563+0000 7f1ec9b80640 1 --2- 172.21.15.60:0/4236564147 >> v2:172.21.15.114:3300/0 conn(0x7f1ec40d2880 0x7f1ec4162f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.563+0000 7f1ec9b80640 1 -- 172.21.15.60:0/4236564147 >> v1:172.21.15.114:6789/0 conn(0x7f1ec41634d0 legacy=0x7f1ec415f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:02.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.563+0000 7f1ec9b80640 1 -- 172.21.15.60:0/4236564147 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec40cefa0 con 0x7f1ec4148be0 2024-05-09T05:01:02.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.564+0000 7f1ec9b80640 1 --2- 172.21.15.60:0/4236564147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ec4148be0 0x7f1ec40cc7b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1ec4164050 tx=0x7f1ebc02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6281a45cd0968586 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:02.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.564+0000 7f1ec8b7e640 1 -- 172.21.15.60:0/4236564147 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc037070 con 0x7f1ec4148be0 2024-05-09T05:01:02.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.564+0000 7f1ec8b7e640 1 -- 172.21.15.60:0/4236564147 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ebc032440 con 0x7f1ec4148be0 2024-05-09T05:01:02.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.564+0000 7f1ec8b7e640 1 -- 172.21.15.60:0/4236564147 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc032810 con 0x7f1ec4148be0 2024-05-09T05:01:02.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.564+0000 7f1ecbde4640 1 -- 172.21.15.60:0/4236564147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ec4148be0 msgr2=0x7f1ec40cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:02.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.564+0000 7f1ecbde4640 1 --2- 172.21.15.60:0/4236564147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ec4148be0 0x7f1ec40cc7b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1ec4164050 tx=0x7f1ebc02a1e0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.564+0000 7f1ecbde4640 1 -- 172.21.15.60:0/4236564147 shutdown_connections 2024-05-09T05:01:02.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.564+0000 7f1ecbde4640 1 --2- 172.21.15.60:0/4236564147 >> v2:172.21.15.114:3300/0 conn(0x7f1ec40d2880 0x7f1ec4162f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.564+0000 7f1ecbde4640 1 --2- 172.21.15.60:0/4236564147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ec4148be0 0x7f1ec40cc7b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.564+0000 7f1ecbde4640 1 -- 172.21.15.60:0/4236564147 >> 172.21.15.60:0/4236564147 conn(0x7f1ec406e580 msgr2=0x7f1ec40709a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:02.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.564+0000 7f1ecbde4640 1 -- 172.21.15.60:0/4236564147 shutdown_connections 2024-05-09T05:01:02.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.564+0000 7f1ecbde4640 1 -- 172.21.15.60:0/4236564147 wait complete. 2024-05-09T05:01:02.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.566+0000 7f1ecbde4640 1 Processor -- start 2024-05-09T05:01:02.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.566+0000 7f1ecbde4640 1 -- start start 2024-05-09T05:01:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.566+0000 7f1ecbde4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec40d2880 0x7f1ec415ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.566+0000 7f1ecbde4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ec4148be0 0x7f1ec413b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.566+0000 7f1ecbde4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ec41634d0 0x7f1ec41602b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.566+0000 7f1ecbde4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec40d2d10 con 0x7f1ec4148be0 2024-05-09T05:01:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.566+0000 7f1ecbde4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec4148fc0 con 0x7f1ec40d2880 2024-05-09T05:01:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.566+0000 7f1ecbde4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1ec40d2e90 con 0x7f1ec41634d0 2024-05-09T05:01:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.566+0000 7f1ec937f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ec4148be0 0x7f1ec413b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.566+0000 7f1ec937f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ec4148be0 0x7f1ec413b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56202/0 (socket says 172.21.15.60:56202) 2024-05-09T05:01:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.566+0000 7f1ec937f640 1 -- 172.21.15.60:0/2157413318 learned_addr learned my addr 172.21.15.60:0/2157413318 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.567+0000 7f1ec937f640 1 -- 172.21.15.60:0/2157413318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ec41634d0 msgr2=0x7f1ec41602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.567+0000 7f1ec937f640 1 --2- 172.21.15.60:0/2157413318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ec41634d0 0x7f1ec41602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.567+0000 7f1ec9b80640 1 --2- 172.21.15.60:0/2157413318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec40d2880 0x7f1ec415ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.567+0000 7f1ec937f640 1 -- 172.21.15.60:0/2157413318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec40d2880 msgr2=0x7f1ec415ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.567+0000 7f1ec937f640 1 --2- 172.21.15.60:0/2157413318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec40d2880 0x7f1ec415ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.567+0000 7f1ec937f640 1 -- 172.21.15.60:0/2157413318 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec413ec30 con 0x7f1ec4148be0 2024-05-09T05:01:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.567+0000 7f1ec937f640 1 --2- 172.21.15.60:0/2157413318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ec4148be0 0x7f1ec413b460 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800e9e0 tx=0x7f1eb800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.567+0000 7f1ec27fc640 1 -- 172.21.15.60:0/2157413318 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8017070 con 0x7f1ec4148be0 2024-05-09T05:01:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.567+0000 7f1ec27fc640 1 -- 172.21.15.60:0/2157413318 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1eb800c970 con 0x7f1ec4148be0 2024-05-09T05:01:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.567+0000 7f1ecbde4640 1 -- 172.21.15.60:0/2157413318 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ec413ef10 con 0x7f1ec4148be0 2024-05-09T05:01:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.567+0000 7f1ec27fc640 1 -- 172.21.15.60:0/2157413318 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb800cd60 con 0x7f1ec4148be0 2024-05-09T05:01:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.567+0000 7f1ecbde4640 1 -- 172.21.15.60:0/2157413318 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ec413f4d0 con 0x7f1ec4148be0 2024-05-09T05:01:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.568+0000 7f1ecbde4640 1 -- 172.21.15.60:0/2157413318 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ec4051040 con 0x7f1ec4148be0 2024-05-09T05:01:02.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.569+0000 7f1ec27fc640 1 -- 172.21.15.60:0/2157413318 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1eb801a070 con 0x7f1ec4148be0 2024-05-09T05:01:02.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.569+0000 7f1ec27fc640 1 --2- 172.21.15.60:0/2157413318 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1eb00762d0 0x7f1eb0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:02.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.569+0000 7f1ec27fc640 1 -- 172.21.15.60:0/2157413318 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5249+0+0 (secure 0 0 0) 0x7f1eb8097830 con 0x7f1ec4148be0 2024-05-09T05:01:02.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.569+0000 7f1ec9b80640 1 --2- 172.21.15.60:0/2157413318 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1eb00762d0 0x7f1eb0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:02.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.570+0000 7f1ec9b80640 1 --2- 172.21.15.60:0/2157413318 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1eb00762d0 0x7f1eb0078790 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc034000 tx=0x7f1ebc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:02.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.573+0000 7f1ec27fc640 1 -- 172.21.15.60:0/2157413318 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1eb80612c0 con 0x7f1ec4148be0 2024-05-09T05:01:02.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:02.768+0000 7f1ecbde4640 1 -- 172.21.15.60:0/2157413318 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f1ec4072cd0 con 0x7f1ec4148be0 2024-05-09T05:01:03.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.316+0000 7f1ec27fc640 1 -- 172.21.15.60:0/2157413318 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v33) ==== 138+0+0 (secure 0 0 0) 0x7f1eb8060bc0 con 0x7f1ec4148be0 2024-05-09T05:01:03.317 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2024-05-09T05:01:03.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.320+0000 7f1ecbde4640 1 -- 172.21.15.60:0/2157413318 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1eb00762d0 msgr2=0x7f1eb0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:03.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.320+0000 7f1ecbde4640 1 --2- 172.21.15.60:0/2157413318 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1eb00762d0 0x7f1eb0078790 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc034000 tx=0x7f1ebc02e000 comp rx=0 tx=0).stop 2024-05-09T05:01:03.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.320+0000 7f1ecbde4640 1 -- 172.21.15.60:0/2157413318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ec4148be0 msgr2=0x7f1ec413b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:03.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.320+0000 7f1ecbde4640 1 --2- 172.21.15.60:0/2157413318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ec4148be0 0x7f1ec413b460 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800e9e0 tx=0x7f1eb800eea0 comp rx=0 tx=0).stop 2024-05-09T05:01:03.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.320+0000 7f1ecbde4640 1 -- 172.21.15.60:0/2157413318 shutdown_connections 2024-05-09T05:01:03.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.320+0000 7f1ecbde4640 1 --2- 172.21.15.60:0/2157413318 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1eb00762d0 0x7f1eb0078790 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc034000 tx=0x7f1ebc02e000 comp rx=0 tx=0).stop 2024-05-09T05:01:03.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.320+0000 7f1ecbde4640 1 --2- 172.21.15.60:0/2157413318 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ec41634d0 0x7f1ec41602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:03.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.320+0000 7f1ecbde4640 1 --2- 172.21.15.60:0/2157413318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ec4148be0 0x7f1ec413b460 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:03.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.320+0000 7f1ecbde4640 1 --2- 172.21.15.60:0/2157413318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ec40d2880 0x7f1ec415ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:03.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.320+0000 7f1ecbde4640 1 -- 172.21.15.60:0/2157413318 >> 172.21.15.60:0/2157413318 conn(0x7f1ec406e580 msgr2=0x7f1ec4144970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:03.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.321+0000 7f1ecbde4640 1 -- 172.21.15.60:0/2157413318 shutdown_connections 2024-05-09T05:01:03.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.321+0000 7f1ecbde4640 1 -- 172.21.15.60:0/2157413318 wait complete. 2024-05-09T05:01:03.336 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-09T05:01:03.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.449+0000 7f34ecf33640 1 Processor -- start 2024-05-09T05:01:03.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.449+0000 7f34ecf33640 1 -- start start 2024-05-09T05:01:03.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.450+0000 7f34ecf33640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f34e815cfd0 0x7f34e8073a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:03.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.450+0000 7f34ecf33640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34e806fcd0 0x7f34e8073f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:03.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.450+0000 7f34ecf33640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34e80d0c60 con 0x7f34e80cf380 2024-05-09T05:01:03.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.450+0000 7f34ecf33640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f34e80d0f00 con 0x7f34e815cfd0 2024-05-09T05:01:03.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.450+0000 7f34ecf33640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f34e8161420 con 0x7f34e806fcd0 2024-05-09T05:01:03.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.450+0000 7f34dffff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34e806fcd0 0x7f34e8073f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:03.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.450+0000 7f34e659c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f34e815cfd0 0x7f34e8073a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:03.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.450+0000 7f34dffff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34e806fcd0 0x7f34e8073f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.60:57914/0 (socket says 172.21.15.60:57914) 2024-05-09T05:01:03.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.450+0000 7f34dffff640 1 -- 172.21.15.60:0/513172001 learned_addr learned my addr 172.21.15.60:0/513172001 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:03.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.451+0000 7f34dffff640 1 -- 172.21.15.60:0/513172001 >> v2:172.21.15.114:3300/0 conn(0x7f34e815cfd0 msgr2=0x7f34e8073a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:03.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.451+0000 7f34dffff640 1 --2- 172.21.15.60:0/513172001 >> v2:172.21.15.114:3300/0 conn(0x7f34e815cfd0 0x7f34e8073a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:03.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.451+0000 7f34dffff640 1 -- 172.21.15.60:0/513172001 >> v1:172.21.15.114:6789/0 conn(0x7f34e80cf380 legacy=0x7f34e806e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:03.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.451+0000 7f34dffff640 1 -- 172.21.15.60:0/513172001 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e8160dd0 con 0x7f34e806fcd0 2024-05-09T05:01:03.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.451+0000 7f34dffff640 1 --2- 172.21.15.60:0/513172001 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34e806fcd0 0x7f34e8073f60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009870 tx=0x7f34d002a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b61d148a85ed9787 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:03.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.452+0000 7f34e5d9b640 1 -- 172.21.15.60:0/513172001 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d0037070 con 0x7f34e806fcd0 2024-05-09T05:01:03.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.452+0000 7f34e5d9b640 1 -- 172.21.15.60:0/513172001 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34d0032440 con 0x7f34e806fcd0 2024-05-09T05:01:03.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.452+0000 7f34e5d9b640 1 -- 172.21.15.60:0/513172001 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d0032810 con 0x7f34e806fcd0 2024-05-09T05:01:03.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.452+0000 7f34ecf33640 1 -- 172.21.15.60:0/513172001 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34e806fcd0 msgr2=0x7f34e8073f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:03.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.452+0000 7f34ecf33640 1 --2- 172.21.15.60:0/513172001 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34e806fcd0 0x7f34e8073f60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009870 tx=0x7f34d002a270 comp rx=0 tx=0).stop 2024-05-09T05:01:03.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.452+0000 7f34ecf33640 1 -- 172.21.15.60:0/513172001 shutdown_connections 2024-05-09T05:01:03.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.452+0000 7f34ecf33640 1 --2- 172.21.15.60:0/513172001 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34e806fcd0 0x7f34e8073f60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:03.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.452+0000 7f34ecf33640 1 --2- 172.21.15.60:0/513172001 >> v2:172.21.15.114:3300/0 conn(0x7f34e815cfd0 0x7f34e8073a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:03.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.452+0000 7f34ecf33640 1 -- 172.21.15.60:0/513172001 >> 172.21.15.60:0/513172001 conn(0x7f34e80b0da0 msgr2=0x7f34e80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:03.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.452+0000 7f34ecf33640 1 -- 172.21.15.60:0/513172001 shutdown_connections 2024-05-09T05:01:03.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.452+0000 7f34ecf33640 1 -- 172.21.15.60:0/513172001 wait complete. 2024-05-09T05:01:03.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.454+0000 7f34ecf33640 1 Processor -- start 2024-05-09T05:01:03.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.454+0000 7f34ecf33640 1 -- start start 2024-05-09T05:01:03.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.454+0000 7f34ecf33640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34e806fcd0 0x7f34e8154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:03.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.454+0000 7f34ecf33640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34e80cf380 0x7f34e815b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:03.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.454+0000 7f34ecf33640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34e815cfd0 0x7f34e8159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:03.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.454+0000 7f34ecf33640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f34e8070390 con 0x7f34e80cf380 2024-05-09T05:01:03.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.454+0000 7f34ecf33640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f34e815e3c0 con 0x7f34e815cfd0 2024-05-09T05:01:03.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.454+0000 7f34ecf33640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f34e8070510 con 0x7f34e806fcd0 2024-05-09T05:01:03.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.455+0000 7f34dffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34e80cf380 0x7f34e815b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:03.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.455+0000 7f34dffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34e80cf380 0x7f34e815b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56214/0 (socket says 172.21.15.60:56214) 2024-05-09T05:01:03.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.455+0000 7f34dffff640 1 -- 172.21.15.60:0/3486317260 learned_addr learned my addr 172.21.15.60:0/3486317260 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:03.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.455+0000 7f34e659c640 1 --2- 172.21.15.60:0/3486317260 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34e806fcd0 0x7f34e8154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:03.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.455+0000 7f34e6d9d640 1 --2- 172.21.15.60:0/3486317260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34e815cfd0 0x7f34e8159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:03.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.455+0000 7f34dffff640 1 -- 172.21.15.60:0/3486317260 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34e806fcd0 msgr2=0x7f34e8154bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:03.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.455+0000 7f34dffff640 1 --2- 172.21.15.60:0/3486317260 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34e806fcd0 0x7f34e8154bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:03.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.455+0000 7f34dffff640 1 -- 172.21.15.60:0/3486317260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34e815cfd0 msgr2=0x7f34e8159410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:03.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.455+0000 7f34dffff640 1 --2- 172.21.15.60:0/3486317260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34e815cfd0 0x7f34e8159410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:03.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.455+0000 7f34dffff640 1 -- 172.21.15.60:0/3486317260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e81539e0 con 0x7f34e80cf380 2024-05-09T05:01:03.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.455+0000 7f34e6d9d640 1 --2- 172.21.15.60:0/3486317260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34e815cfd0 0x7f34e8159410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:03.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.455+0000 7f34dffff640 1 --2- 172.21.15.60:0/3486317260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34e80cf380 0x7f34e815b2f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f34d002a2b0 tx=0x7f34d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:03.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.455+0000 7f34deffd640 1 -- 172.21.15.60:0/3486317260 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d0037040 con 0x7f34e80cf380 2024-05-09T05:01:03.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.455+0000 7f34deffd640 1 -- 172.21.15.60:0/3486317260 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34d0042070 con 0x7f34e80cf380 2024-05-09T05:01:03.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.455+0000 7f34ecf33640 1 -- 172.21.15.60:0/3486317260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34e8152dd0 con 0x7f34e80cf380 2024-05-09T05:01:03.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.455+0000 7f34deffd640 1 -- 172.21.15.60:0/3486317260 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d0030860 con 0x7f34e80cf380 2024-05-09T05:01:03.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.456+0000 7f34ecf33640 1 -- 172.21.15.60:0/3486317260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34e8153390 con 0x7f34e80cf380 2024-05-09T05:01:03.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.456+0000 7f34ecf33640 1 -- 172.21.15.60:0/3486317260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34e8051040 con 0x7f34e80cf380 2024-05-09T05:01:03.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.457+0000 7f34deffd640 1 -- 172.21.15.60:0/3486317260 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f34d0030a00 con 0x7f34e80cf380 2024-05-09T05:01:03.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.458+0000 7f34deffd640 1 --2- 172.21.15.60:0/3486317260 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f34c80762d0 0x7f34c8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:03.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.458+0000 7f34deffd640 1 -- 172.21.15.60:0/3486317260 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5619+0+0 (secure 0 0 0) 0x7f34d00b7e10 con 0x7f34e80cf380 2024-05-09T05:01:03.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.458+0000 7f34e659c640 1 --2- 172.21.15.60:0/3486317260 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f34c80762d0 0x7f34c8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:03.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.459+0000 7f34e659c640 1 --2- 172.21.15.60:0/3486317260 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f34c80762d0 0x7f34c8078790 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009aa0 tx=0x7f34d8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:03.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.463+0000 7f34deffd640 1 -- 172.21.15.60:0/3486317260 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f34d0081740 con 0x7f34e80cf380 2024-05-09T05:01:03.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:03.653+0000 7f34ecf33640 1 -- 172.21.15.60:0/3486317260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f34e8159170 con 0x7f34e80cf380 2024-05-09T05:01:04.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.319+0000 7f34deffd640 1 -- 172.21.15.60:0/3486317260 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v34) ==== 170+0+0 (secure 0 0 0) 0x7f34d0081040 con 0x7f34e80cf380 2024-05-09T05:01:04.320 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2024-05-09T05:01:04.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.321+0000 7f34ecf33640 1 -- 172.21.15.60:0/3486317260 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f34c80762d0 msgr2=0x7f34c8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:04.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.321+0000 7f34ecf33640 1 --2- 172.21.15.60:0/3486317260 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f34c80762d0 0x7f34c8078790 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009aa0 tx=0x7f34d8009190 comp rx=0 tx=0).stop 2024-05-09T05:01:04.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.321+0000 7f34ecf33640 1 -- 172.21.15.60:0/3486317260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34e80cf380 msgr2=0x7f34e815b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:04.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.321+0000 7f34ecf33640 1 --2- 172.21.15.60:0/3486317260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34e80cf380 0x7f34e815b2f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f34d002a2b0 tx=0x7f34d00046c0 comp rx=0 tx=0).stop 2024-05-09T05:01:04.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.321+0000 7f34ecf33640 1 -- 172.21.15.60:0/3486317260 shutdown_connections 2024-05-09T05:01:04.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.321+0000 7f34ecf33640 1 --2- 172.21.15.60:0/3486317260 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f34c80762d0 0x7f34c8078790 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:04.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.321+0000 7f34ecf33640 1 --2- 172.21.15.60:0/3486317260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34e815cfd0 0x7f34e8159410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:04.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.321+0000 7f34ecf33640 1 --2- 172.21.15.60:0/3486317260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34e80cf380 0x7f34e815b2f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:04.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.321+0000 7f34ecf33640 1 --2- 172.21.15.60:0/3486317260 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f34e806fcd0 0x7f34e8154bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:04.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.321+0000 7f34ecf33640 1 -- 172.21.15.60:0/3486317260 >> 172.21.15.60:0/3486317260 conn(0x7f34e80b0da0 msgr2=0x7f34e80b1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:04.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.321+0000 7f34ecf33640 1 -- 172.21.15.60:0/3486317260 shutdown_connections 2024-05-09T05:01:04.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.321+0000 7f34ecf33640 1 -- 172.21.15.60:0/3486317260 wait complete. 2024-05-09T05:01:04.347 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-09T05:01:04.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.456+0000 7f333123f640 1 Processor -- start 2024-05-09T05:01:04.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.456+0000 7f333123f640 1 -- start start 2024-05-09T05:01:04.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.457+0000 7f333123f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f332c0b7080 0x7f332c0b7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:04.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.457+0000 7f333123f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f332c0d33a0 con 0x7f332c1486d0 2024-05-09T05:01:04.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.457+0000 7f333123f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f332c0b79d0 con 0x7f332c148c40 2024-05-09T05:01:04.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.457+0000 7f333123f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f332c071810 con 0x7f332c0b7080 2024-05-09T05:01:04.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.457+0000 7f332b59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f332c0b7080 0x7f332c0b7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:04.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.457+0000 7f332ad9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f332c1486d0 0x7f332c0b5ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51504/0 (socket says 172.21.15.60:51504) 2024-05-09T05:01:04.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.457+0000 7f332b59e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f332c0b7080 0x7f332c0b7490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56218/0 (socket says 172.21.15.60:56218) 2024-05-09T05:01:04.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.457+0000 7f332ad9d640 1 -- 172.21.15.60:0/1147634543 learned_addr learned my addr 172.21.15.60:0/1147634543 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:04.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.457+0000 7f332b59e640 1 -- 172.21.15.60:0/1147634543 >> v1:172.21.15.114:6789/0 conn(0x7f332c148c40 legacy=0x7f332c0d2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:04.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.457+0000 7f332b59e640 1 -- 172.21.15.60:0/1147634543 >> v1:172.21.15.60:6789/0 conn(0x7f332c1486d0 legacy=0x7f332c0b5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:04.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.457+0000 7f332b59e640 1 -- 172.21.15.60:0/1147634543 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f332c071a60 con 0x7f332c0b7080 2024-05-09T05:01:04.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.458+0000 7f332b59e640 1 --2- 172.21.15.60:0/1147634543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f332c0b7080 0x7f332c0b7490 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f331c009d90 tx=0x7f331c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f368a5e9692a3dcf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:04.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.458+0000 7f3329d9b640 1 -- 172.21.15.60:0/1147634543 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f331c007740 con 0x7f332c0b7080 2024-05-09T05:01:04.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.458+0000 7f3329d9b640 1 -- 172.21.15.60:0/1147634543 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f331c0078e0 con 0x7f332c0b7080 2024-05-09T05:01:04.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.458+0000 7f3329d9b640 1 -- 172.21.15.60:0/1147634543 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f331c007cb0 con 0x7f332c0b7080 2024-05-09T05:01:04.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.459+0000 7f333123f640 1 -- 172.21.15.60:0/1147634543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f332c0b7080 msgr2=0x7f332c0b7490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:04.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.459+0000 7f333123f640 1 --2- 172.21.15.60:0/1147634543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f332c0b7080 0x7f332c0b7490 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f331c009d90 tx=0x7f331c02a6f0 comp rx=0 tx=0).stop 2024-05-09T05:01:04.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.459+0000 7f333123f640 1 -- 172.21.15.60:0/1147634543 shutdown_connections 2024-05-09T05:01:04.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.459+0000 7f333123f640 1 --2- 172.21.15.60:0/1147634543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f332c0b7080 0x7f332c0b7490 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:04.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.459+0000 7f333123f640 1 -- 172.21.15.60:0/1147634543 >> 172.21.15.60:0/1147634543 conn(0x7f332c06f6f0 msgr2=0x7f332c0ba5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:04.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.459+0000 7f333123f640 1 -- 172.21.15.60:0/1147634543 shutdown_connections 2024-05-09T05:01:04.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.459+0000 7f333123f640 1 -- 172.21.15.60:0/1147634543 wait complete. 2024-05-09T05:01:04.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.460+0000 7f333123f640 1 Processor -- start 2024-05-09T05:01:04.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.460+0000 7f333123f640 1 -- start start 2024-05-09T05:01:04.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.461+0000 7f333123f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f332c0b7080 0x7f332c15edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:04.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.461+0000 7f333123f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f332c1486d0 0x7f332c162140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:04.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.461+0000 7f333123f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f332c148c40 0x7f332c15f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:04.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.461+0000 7f333123f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f332c072300 con 0x7f332c1486d0 2024-05-09T05:01:04.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.461+0000 7f333123f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f332c072020 con 0x7f332c148c40 2024-05-09T05:01:04.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.461+0000 7f333123f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f332c072820 con 0x7f332c0b7080 2024-05-09T05:01:04.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.461+0000 7f332a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f332c1486d0 0x7f332c162140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:04.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.461+0000 7f332a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f332c1486d0 0x7f332c162140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56226/0 (socket says 172.21.15.60:56226) 2024-05-09T05:01:04.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.461+0000 7f332a59c640 1 -- 172.21.15.60:0/101340193 learned_addr learned my addr 172.21.15.60:0/101340193 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:04.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.461+0000 7f332ad9d640 1 --2- 172.21.15.60:0/101340193 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f332c0b7080 0x7f332c15edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:04.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.461+0000 7f332b59e640 1 --2- 172.21.15.60:0/101340193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f332c148c40 0x7f332c15f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:04.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.462+0000 7f332a59c640 1 -- 172.21.15.60:0/101340193 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f332c0b7080 msgr2=0x7f332c15edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:04.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.462+0000 7f332a59c640 1 --2- 172.21.15.60:0/101340193 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f332c0b7080 0x7f332c15edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:04.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.462+0000 7f332a59c640 1 -- 172.21.15.60:0/101340193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f332c148c40 msgr2=0x7f332c15f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:04.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.462+0000 7f332a59c640 1 --2- 172.21.15.60:0/101340193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f332c148c40 0x7f332c15f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:04.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.462+0000 7f332a59c640 1 -- 172.21.15.60:0/101340193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f332c15f310 con 0x7f332c1486d0 2024-05-09T05:01:04.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.462+0000 7f332b59e640 1 --2- 172.21.15.60:0/101340193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f332c148c40 0x7f332c15f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:04.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.462+0000 7f332a59c640 1 --2- 172.21.15.60:0/101340193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f332c1486d0 0x7f332c162140 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f331400b3e0 tx=0x7f331400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:04.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.462+0000 7f33237fe640 1 -- 172.21.15.60:0/101340193 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3314015020 con 0x7f332c1486d0 2024-05-09T05:01:04.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.462+0000 7f33237fe640 1 -- 172.21.15.60:0/101340193 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3314011850 con 0x7f332c1486d0 2024-05-09T05:01:04.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.462+0000 7f333123f640 1 -- 172.21.15.60:0/101340193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f332c13ecd0 con 0x7f332c1486d0 2024-05-09T05:01:04.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.462+0000 7f33237fe640 1 -- 172.21.15.60:0/101340193 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3314011c40 con 0x7f332c1486d0 2024-05-09T05:01:04.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.462+0000 7f333123f640 1 -- 172.21.15.60:0/101340193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f332c13f290 con 0x7f332c1486d0 2024-05-09T05:01:04.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.463+0000 7f333123f640 1 -- 172.21.15.60:0/101340193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f332c051040 con 0x7f332c1486d0 2024-05-09T05:01:04.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.464+0000 7f33237fe640 1 -- 172.21.15.60:0/101340193 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3314018020 con 0x7f332c1486d0 2024-05-09T05:01:04.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.464+0000 7f33237fe640 1 --2- 172.21.15.60:0/101340193 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f330c0762d0 0x7f330c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:04.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.464+0000 7f33237fe640 1 -- 172.21.15.60:0/101340193 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6001+0+0 (secure 0 0 0) 0x7f3314097550 con 0x7f332c1486d0 2024-05-09T05:01:04.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.464+0000 7f332ad9d640 1 --2- 172.21.15.60:0/101340193 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f330c0762d0 0x7f330c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:04.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.465+0000 7f332ad9d640 1 --2- 172.21.15.60:0/101340193 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f330c0762d0 0x7f330c078790 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3324000f60 tx=0x7f3324009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:04.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.469+0000 7f33237fe640 1 -- 172.21.15.60:0/101340193 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f331401f020 con 0x7f332c1486d0 2024-05-09T05:01:04.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:04.697+0000 7f333123f640 1 -- 172.21.15.60:0/101340193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f332c073980 con 0x7f332c1486d0 2024-05-09T05:01:05.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.337+0000 7f33237fe640 1 -- 172.21.15.60:0/101340193 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) ==== 172+0+0 (secure 0 0 0) 0x7f332c15f310 con 0x7f332c1486d0 2024-05-09T05:01:05.342 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 5 and data pool 6 2024-05-09T05:01:05.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.344+0000 7f333123f640 1 -- 172.21.15.60:0/101340193 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f330c0762d0 msgr2=0x7f330c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:05.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.344+0000 7f333123f640 1 --2- 172.21.15.60:0/101340193 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f330c0762d0 0x7f330c078790 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3324000f60 tx=0x7f3324009040 comp rx=0 tx=0).stop 2024-05-09T05:01:05.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.344+0000 7f333123f640 1 -- 172.21.15.60:0/101340193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f332c1486d0 msgr2=0x7f332c162140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:05.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.344+0000 7f333123f640 1 --2- 172.21.15.60:0/101340193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f332c1486d0 0x7f332c162140 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f331400b3e0 tx=0x7f331400b8a0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.344+0000 7f333123f640 1 -- 172.21.15.60:0/101340193 shutdown_connections 2024-05-09T05:01:05.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.344+0000 7f333123f640 1 --2- 172.21.15.60:0/101340193 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f330c0762d0 0x7f330c078790 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.344+0000 7f333123f640 1 --2- 172.21.15.60:0/101340193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f332c148c40 0x7f332c15f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.345+0000 7f333123f640 1 --2- 172.21.15.60:0/101340193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f332c1486d0 0x7f332c162140 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.345+0000 7f333123f640 1 --2- 172.21.15.60:0/101340193 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f332c0b7080 0x7f332c15edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.345+0000 7f333123f640 1 -- 172.21.15.60:0/101340193 >> 172.21.15.60:0/101340193 conn(0x7f332c06f6f0 msgr2=0x7f332c0b8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:05.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.345+0000 7f333123f640 1 -- 172.21.15.60:0/101340193 shutdown_connections 2024-05-09T05:01:05.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.346+0000 7f333123f640 1 -- 172.21.15.60:0/101340193 wait complete. 2024-05-09T05:01:05.371 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:01:05.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.479+0000 7f62f54ea640 1 Processor -- start 2024-05-09T05:01:05.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.479+0000 7f62f54ea640 1 -- start start 2024-05-09T05:01:05.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.480+0000 7f62f54ea640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f62f015f490 0x7f62f006fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:05.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.480+0000 7f62f54ea640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62f00d2b50 0x7f62f0072850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:05.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.480+0000 7f62f54ea640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62f006f6a0 con 0x7f62f0070080 2024-05-09T05:01:05.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.480+0000 7f62f54ea640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f62f00704e0 con 0x7f62f015f490 2024-05-09T05:01:05.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.480+0000 7f62f54ea640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f62f006e8e0 con 0x7f62f00d2b50 2024-05-09T05:01:05.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.480+0000 7f62eeffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f62f015f490 0x7f62f006fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:05.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.480+0000 7f62eeffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f62f015f490 0x7f62f006fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56238/0 (socket says 172.21.15.60:56238) 2024-05-09T05:01:05.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.480+0000 7f62eeffd640 1 -- 172.21.15.60:0/4289015828 learned_addr learned my addr 172.21.15.60:0/4289015828 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:05.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.480+0000 7f62ef7fe640 1 --2- 172.21.15.60:0/4289015828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62f00d2b50 0x7f62f0072850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:05.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.480+0000 7f62eeffd640 1 -- 172.21.15.60:0/4289015828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62f00d2b50 msgr2=0x7f62f0072850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:05.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.480+0000 7f62eeffd640 1 --2- 172.21.15.60:0/4289015828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62f00d2b50 0x7f62f0072850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.481+0000 7f62eeffd640 1 -- 172.21.15.60:0/4289015828 >> v1:172.21.15.114:6789/0 conn(0x7f62f0070080 legacy=0x7f62f00d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:05.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.481+0000 7f62eeffd640 1 -- 172.21.15.60:0/4289015828 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62f006ebc0 con 0x7f62f015f490 2024-05-09T05:01:05.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.481+0000 7f62eeffd640 1 --2- 172.21.15.60:0/4289015828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62f015f490 0x7f62f006fb40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009870 tx=0x7f62e002a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=362240ab2d00fa8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:05.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.481+0000 7f62edffb640 1 -- 172.21.15.60:0/4289015828 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e0037070 con 0x7f62f015f490 2024-05-09T05:01:05.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.481+0000 7f62edffb640 1 -- 172.21.15.60:0/4289015828 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62e0032440 con 0x7f62f015f490 2024-05-09T05:01:05.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.481+0000 7f62edffb640 1 -- 172.21.15.60:0/4289015828 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e0032810 con 0x7f62f015f490 2024-05-09T05:01:05.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.482+0000 7f62f54ea640 1 -- 172.21.15.60:0/4289015828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62f015f490 msgr2=0x7f62f006fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:05.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.482+0000 7f62f54ea640 1 --2- 172.21.15.60:0/4289015828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62f015f490 0x7f62f006fb40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009870 tx=0x7f62e002a130 comp rx=0 tx=0).stop 2024-05-09T05:01:05.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.482+0000 7f62f54ea640 1 -- 172.21.15.60:0/4289015828 shutdown_connections 2024-05-09T05:01:05.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.482+0000 7f62f54ea640 1 --2- 172.21.15.60:0/4289015828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62f00d2b50 0x7f62f0072850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.482+0000 7f62f54ea640 1 --2- 172.21.15.60:0/4289015828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62f015f490 0x7f62f006fb40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.482+0000 7f62f54ea640 1 -- 172.21.15.60:0/4289015828 >> 172.21.15.60:0/4289015828 conn(0x7f62f009b8c0 msgr2=0x7f62f009bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:05.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.482+0000 7f62f54ea640 1 -- 172.21.15.60:0/4289015828 shutdown_connections 2024-05-09T05:01:05.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.482+0000 7f62f54ea640 1 -- 172.21.15.60:0/4289015828 wait complete. 2024-05-09T05:01:05.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.483+0000 7f62f54ea640 1 Processor -- start 2024-05-09T05:01:05.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.483+0000 7f62f54ea640 1 -- start start 2024-05-09T05:01:05.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.484+0000 7f62f54ea640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62f0070080 0x7f62f0151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:05.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.484+0000 7f62f54ea640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62f00d2b50 0x7f62f01e8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:05.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.484+0000 7f62f54ea640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62f015f490 0x7f62f01eac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:05.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.484+0000 7f62f54ea640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f62f0072d90 con 0x7f62f00d2b50 2024-05-09T05:01:05.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.484+0000 7f62f54ea640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f62f015f870 con 0x7f62f0070080 2024-05-09T05:01:05.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.484+0000 7f62f54ea640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f62f0072f10 con 0x7f62f015f490 2024-05-09T05:01:05.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.484+0000 7f62ee7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62f00d2b50 0x7f62f01e8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:05.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.484+0000 7f62ee7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62f00d2b50 0x7f62f01e8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56240/0 (socket says 172.21.15.60:56240) 2024-05-09T05:01:05.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.484+0000 7f62ee7fc640 1 -- 172.21.15.60:0/808209231 learned_addr learned my addr 172.21.15.60:0/808209231 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:05.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.484+0000 7f62eeffd640 1 --2- 172.21.15.60:0/808209231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62f0070080 0x7f62f0151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:05.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.484+0000 7f62ef7fe640 1 --2- 172.21.15.60:0/808209231 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62f015f490 0x7f62f01eac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:05.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.485+0000 7f62ee7fc640 1 -- 172.21.15.60:0/808209231 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62f015f490 msgr2=0x7f62f01eac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:05.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.485+0000 7f62ee7fc640 1 --2- 172.21.15.60:0/808209231 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62f015f490 0x7f62f01eac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.485+0000 7f62ee7fc640 1 -- 172.21.15.60:0/808209231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62f0070080 msgr2=0x7f62f0151070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:05.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.485+0000 7f62ee7fc640 1 --2- 172.21.15.60:0/808209231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62f0070080 0x7f62f0151070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.485+0000 7f62ee7fc640 1 -- 172.21.15.60:0/808209231 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62f01e93f0 con 0x7f62f00d2b50 2024-05-09T05:01:05.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.485+0000 7f62ee7fc640 1 --2- 172.21.15.60:0/808209231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62f00d2b50 0x7f62f01e8eb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f62d800c430 tx=0x7f62d800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:05.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.485+0000 7f62e77fe640 1 -- 172.21.15.60:0/808209231 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8011790 con 0x7f62f00d2b50 2024-05-09T05:01:05.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.485+0000 7f62e77fe640 1 -- 172.21.15.60:0/808209231 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62d8017070 con 0x7f62f00d2b50 2024-05-09T05:01:05.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.485+0000 7f62f54ea640 1 -- 172.21.15.60:0/808209231 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62f01e96c0 con 0x7f62f00d2b50 2024-05-09T05:01:05.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.485+0000 7f62e77fe640 1 -- 172.21.15.60:0/808209231 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8011b70 con 0x7f62f00d2b50 2024-05-09T05:01:05.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.485+0000 7f62f54ea640 1 -- 172.21.15.60:0/808209231 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62f01e9c80 con 0x7f62f00d2b50 2024-05-09T05:01:05.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.486+0000 7f62f54ea640 1 -- 172.21.15.60:0/808209231 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62f0051040 con 0x7f62f00d2b50 2024-05-09T05:01:05.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.487+0000 7f62e77fe640 1 -- 172.21.15.60:0/808209231 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f62d8011d10 con 0x7f62f00d2b50 2024-05-09T05:01:05.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.487+0000 7f62e77fe640 1 --2- 172.21.15.60:0/808209231 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f62d00762d0 0x7f62d0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:05.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.487+0000 7f62e77fe640 1 -- 172.21.15.60:0/808209231 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6101+0+0 (secure 0 0 0) 0x7f62d80974a0 con 0x7f62f00d2b50 2024-05-09T05:01:05.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.487+0000 7f62eeffd640 1 --2- 172.21.15.60:0/808209231 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f62d00762d0 0x7f62d0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:05.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.488+0000 7f62eeffd640 1 --2- 172.21.15.60:0/808209231 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f62d00762d0 0x7f62d0078790 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f62e0034000 tx=0x7f62e002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:05.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.492+0000 7f62e77fe640 1 -- 172.21.15.60:0/808209231 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f62d8060620 con 0x7f62f00d2b50 2024-05-09T05:01:05.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.686+0000 7f62f54ea640 1 -- 172.21.15.60:0/808209231 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f62f0099070 con 0x7f62f00d2b50 2024-05-09T05:01:05.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.687+0000 7f62e77fe640 1 -- 172.21.15.60:0/808209231 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+16931 (secure 0 0 0) 0x7f62d8064920 con 0x7f62f00d2b50 2024-05-09T05:01:05.688 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:05.688 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":35,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:01:04.699010+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:01:02.773384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-05-09T05:01:03.658182+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:01:05.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.690+0000 7f62f54ea640 1 -- 172.21.15.60:0/808209231 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f62d00762d0 msgr2=0x7f62d0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:05.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.690+0000 7f62f54ea640 1 --2- 172.21.15.60:0/808209231 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f62d00762d0 0x7f62d0078790 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f62e0034000 tx=0x7f62e002e000 comp rx=0 tx=0).stop 2024-05-09T05:01:05.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.690+0000 7f62f54ea640 1 -- 172.21.15.60:0/808209231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62f00d2b50 msgr2=0x7f62f01e8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:05.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.690+0000 7f62f54ea640 1 --2- 172.21.15.60:0/808209231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62f00d2b50 0x7f62f01e8eb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f62d800c430 tx=0x7f62d800c8f0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.690+0000 7f62f54ea640 1 -- 172.21.15.60:0/808209231 shutdown_connections 2024-05-09T05:01:05.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.690+0000 7f62f54ea640 1 --2- 172.21.15.60:0/808209231 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f62d00762d0 0x7f62d0078790 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.690+0000 7f62f54ea640 1 --2- 172.21.15.60:0/808209231 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f62f015f490 0x7f62f01eac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.690+0000 7f62f54ea640 1 --2- 172.21.15.60:0/808209231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62f00d2b50 0x7f62f01e8eb0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.690+0000 7f62f54ea640 1 --2- 172.21.15.60:0/808209231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62f0070080 0x7f62f0151070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.690+0000 7f62f54ea640 1 -- 172.21.15.60:0/808209231 >> 172.21.15.60:0/808209231 conn(0x7f62f009b8c0 msgr2=0x7f62f009d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:05.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.690+0000 7f62f54ea640 1 -- 172.21.15.60:0/808209231 shutdown_connections 2024-05-09T05:01:05.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.690+0000 7f62f54ea640 1 -- 172.21.15.60:0/808209231 wait complete. 2024-05-09T05:01:05.702 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:01:05.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.805+0000 7f558a94e640 1 Processor -- start 2024-05-09T05:01:05.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.805+0000 7f558a94e640 1 -- start start 2024-05-09T05:01:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.806+0000 7f558a94e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415f300 0x7f5584072220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.806+0000 7f558a94e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5584073de0 0x7f55840702a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.806+0000 7f558a94e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5584072760 con 0x7f558406fe20 2024-05-09T05:01:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.806+0000 7f558a94e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f5584073240 con 0x7f5584073de0 2024-05-09T05:01:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.806+0000 7f558a94e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f55840d08e0 con 0x7f558415f300 2024-05-09T05:01:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.806+0000 7f5588eeb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5584073de0 0x7f55840702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.806+0000 7f5588eeb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5584073de0 0x7f55840702a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56252/0 (socket says 172.21.15.60:56252) 2024-05-09T05:01:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.806+0000 7f5588eeb640 1 -- 172.21.15.60:0/4248099372 learned_addr learned my addr 172.21.15.60:0/4248099372 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.806+0000 7f5583fff640 1 --2- 172.21.15.60:0/4248099372 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415f300 0x7f5584072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.806+0000 7f5588eeb640 1 -- 172.21.15.60:0/4248099372 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415f300 msgr2=0x7f5584072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.806+0000 7f5588eeb640 1 --2- 172.21.15.60:0/4248099372 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415f300 0x7f5584072220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.806+0000 7f5588eeb640 1 -- 172.21.15.60:0/4248099372 >> v1:172.21.15.114:6789/0 conn(0x7f558406fe20 legacy=0x7f55840736d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.806+0000 7f5588eeb640 1 -- 172.21.15.60:0/4248099372 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55840d0bc0 con 0x7f5584073de0 2024-05-09T05:01:05.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.807+0000 7f5588eeb640 1 --2- 172.21.15.60:0/4248099372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5584073de0 0x7f55840702a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f557c009e50 tx=0x7f557c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f978f923071a6a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:05.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.807+0000 7f5582ffd640 1 -- 172.21.15.60:0/4248099372 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557c036070 con 0x7f5584073de0 2024-05-09T05:01:05.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.807+0000 7f5582ffd640 1 -- 172.21.15.60:0/4248099372 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f557c031600 con 0x7f5584073de0 2024-05-09T05:01:05.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.807+0000 7f5582ffd640 1 -- 172.21.15.60:0/4248099372 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557c0319d0 con 0x7f5584073de0 2024-05-09T05:01:05.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.808+0000 7f558a94e640 1 -- 172.21.15.60:0/4248099372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5584073de0 msgr2=0x7f55840702a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:05.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.808+0000 7f558a94e640 1 --2- 172.21.15.60:0/4248099372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5584073de0 0x7f55840702a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f557c009e50 tx=0x7f557c02a680 comp rx=0 tx=0).stop 2024-05-09T05:01:05.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.808+0000 7f558a94e640 1 -- 172.21.15.60:0/4248099372 shutdown_connections 2024-05-09T05:01:05.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.808+0000 7f558a94e640 1 --2- 172.21.15.60:0/4248099372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5584073de0 0x7f55840702a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.808+0000 7f558a94e640 1 --2- 172.21.15.60:0/4248099372 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415f300 0x7f5584072220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.808+0000 7f558a94e640 1 -- 172.21.15.60:0/4248099372 >> 172.21.15.60:0/4248099372 conn(0x7f55840b4ef0 msgr2=0x7f558406e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:05.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.808+0000 7f558a94e640 1 -- 172.21.15.60:0/4248099372 shutdown_connections 2024-05-09T05:01:05.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.808+0000 7f558a94e640 1 -- 172.21.15.60:0/4248099372 wait complete. 2024-05-09T05:01:05.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.809+0000 7f558a94e640 1 Processor -- start 2024-05-09T05:01:05.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.809+0000 7f558a94e640 1 -- start start 2024-05-09T05:01:05.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.810+0000 7f558a94e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f558406fe20 0x7f558415d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:05.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.810+0000 7f558a94e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5584073de0 0x7f55841e9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:05.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.810+0000 7f558a94e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415f300 0x7f55841e9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:05.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.810+0000 7f558a94e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f55840d04f0 con 0x7f558406fe20 2024-05-09T05:01:05.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.810+0000 7f558a94e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f558409c1e0 con 0x7f5584073de0 2024-05-09T05:01:05.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.810+0000 7f558a94e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f55840d0670 con 0x7f558415f300 2024-05-09T05:01:05.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.810+0000 7f5583fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f558406fe20 0x7f558415d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:05.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.810+0000 7f5583fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f558406fe20 0x7f558415d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56264/0 (socket says 172.21.15.60:56264) 2024-05-09T05:01:05.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.810+0000 7f5583fff640 1 -- 172.21.15.60:0/1399591845 learned_addr learned my addr 172.21.15.60:0/1399591845 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:05.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.810+0000 7f55837fe640 1 --2- 172.21.15.60:0/1399591845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5584073de0 0x7f55841e9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:05.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.810+0000 7f5588eeb640 1 --2- 172.21.15.60:0/1399591845 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415f300 0x7f55841e9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:05.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.810+0000 7f5583fff640 1 -- 172.21.15.60:0/1399591845 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415f300 msgr2=0x7f55841e9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:05.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.810+0000 7f5583fff640 1 --2- 172.21.15.60:0/1399591845 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415f300 0x7f55841e9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.810+0000 7f5583fff640 1 -- 172.21.15.60:0/1399591845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5584073de0 msgr2=0x7f55841e9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:05.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.810+0000 7f5583fff640 1 --2- 172.21.15.60:0/1399591845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5584073de0 0x7f55841e9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:05.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.811+0000 7f5583fff640 1 -- 172.21.15.60:0/1399591845 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55841e8660 con 0x7f558406fe20 2024-05-09T05:01:05.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.811+0000 7f5583fff640 1 --2- 172.21.15.60:0/1399591845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f558406fe20 0x7f558415d080 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f557400d620 tx=0x7f557400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:05.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.811+0000 7f5580ff9640 1 -- 172.21.15.60:0/1399591845 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557400b620 con 0x7f558406fe20 2024-05-09T05:01:05.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.811+0000 7f5580ff9640 1 -- 172.21.15.60:0/1399591845 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5574019070 con 0x7f558406fe20 2024-05-09T05:01:05.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.811+0000 7f5580ff9640 1 -- 172.21.15.60:0/1399591845 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557400ba00 con 0x7f558406fe20 2024-05-09T05:01:05.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.811+0000 7f558a94e640 1 -- 172.21.15.60:0/1399591845 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55841e8940 con 0x7f558406fe20 2024-05-09T05:01:05.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.811+0000 7f558a94e640 1 -- 172.21.15.60:0/1399591845 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55841e8f00 con 0x7f558406fe20 2024-05-09T05:01:05.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.812+0000 7f558a94e640 1 -- 172.21.15.60:0/1399591845 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5584051040 con 0x7f558406fe20 2024-05-09T05:01:05.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.812+0000 7f5580ff9640 1 -- 172.21.15.60:0/1399591845 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f557400bba0 con 0x7f558406fe20 2024-05-09T05:01:05.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.813+0000 7f5580ff9640 1 --2- 172.21.15.60:0/1399591845 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f55700762d0 0x7f5570078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:05.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.813+0000 7f5580ff9640 1 -- 172.21.15.60:0/1399591845 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6101+0+0 (secure 0 0 0) 0x7f5574097c30 con 0x7f558406fe20 2024-05-09T05:01:05.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.813+0000 7f55837fe640 1 --2- 172.21.15.60:0/1399591845 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f55700762d0 0x7f5570078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:05.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.814+0000 7f55837fe640 1 --2- 172.21.15.60:0/1399591845 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f55700762d0 0x7f5570078790 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f558415e2a0 tx=0x7f5578009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:05.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:05.818+0000 7f5580ff9640 1 -- 172.21.15.60:0/1399591845 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5574061400 con 0x7f558406fe20 2024-05-09T05:01:06.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.011+0000 7f558a94e640 1 -- 172.21.15.60:0/1399591845 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f55840b3a00 con 0x7f558406fe20 2024-05-09T05:01:06.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.012+0000 7f5580ff9640 1 -- 172.21.15.60:0/1399591845 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+16931 (secure 0 0 0) 0x7f5574060d00 con 0x7f558406fe20 2024-05-09T05:01:06.013 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:06.013 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":35,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:01:04.699010+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:01:02.773384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-05-09T05:01:03.658182+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:01:06.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.014+0000 7f558a94e640 1 -- 172.21.15.60:0/1399591845 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f55700762d0 msgr2=0x7f5570078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:06.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.014+0000 7f558a94e640 1 --2- 172.21.15.60:0/1399591845 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f55700762d0 0x7f5570078790 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f558415e2a0 tx=0x7f5578009040 comp rx=0 tx=0).stop 2024-05-09T05:01:06.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.014+0000 7f558a94e640 1 -- 172.21.15.60:0/1399591845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f558406fe20 msgr2=0x7f558415d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:06.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.014+0000 7f558a94e640 1 --2- 172.21.15.60:0/1399591845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f558406fe20 0x7f558415d080 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f557400d620 tx=0x7f557400dae0 comp rx=0 tx=0).stop 2024-05-09T05:01:06.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.014+0000 7f558a94e640 1 -- 172.21.15.60:0/1399591845 shutdown_connections 2024-05-09T05:01:06.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.014+0000 7f558a94e640 1 --2- 172.21.15.60:0/1399591845 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f55700762d0 0x7f5570078790 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:06.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.014+0000 7f558a94e640 1 --2- 172.21.15.60:0/1399591845 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f558415f300 0x7f55841e9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:06.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.014+0000 7f558a94e640 1 --2- 172.21.15.60:0/1399591845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5584073de0 0x7f55841e9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:06.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.014+0000 7f558a94e640 1 --2- 172.21.15.60:0/1399591845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f558406fe20 0x7f558415d080 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:06.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.015+0000 7f558a94e640 1 -- 172.21.15.60:0/1399591845 >> 172.21.15.60:0/1399591845 conn(0x7f55840b4ef0 msgr2=0x7f55840b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:06.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.015+0000 7f558a94e640 1 -- 172.21.15.60:0/1399591845 shutdown_connections 2024-05-09T05:01:06.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.015+0000 7f558a94e640 1 -- 172.21.15.60:0/1399591845 wait complete. 2024-05-09T05:01:06.027 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-09T05:01:06.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.133+0000 7f4b7cbc1640 1 Processor -- start 2024-05-09T05:01:06.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.134+0000 7f4b7cbc1640 1 -- start start 2024-05-09T05:01:06.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.134+0000 7f4b7cbc1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b78148c40 0x7f4b780d2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:06.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.134+0000 7f4b7cbc1640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b78071a60 con 0x7f4b780b7080 2024-05-09T05:01:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.134+0000 7f4b7cbc1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b78071d40 con 0x7f4b781486d0 2024-05-09T05:01:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.134+0000 7f4b7cbc1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4b78071810 con 0x7f4b78148c40 2024-05-09T05:01:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b76d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4b780b7080 0x7f4b78072820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51516/0 (socket says 172.21.15.60:51516) 2024-05-09T05:01:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b76d9d640 1 -- 172.21.15.60:0/2893214911 learned_addr learned my addr 172.21.15.60:0/2893214911 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b75d9b640 1 -- 172.21.15.60:0/2893214911 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 273039383 0 0) 0x7f4b78071a60 con 0x7f4b780b7080 2024-05-09T05:01:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b75d9b640 1 -- 172.21.15.60:0/2893214911 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b64003520 con 0x7f4b780b7080 2024-05-09T05:01:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b6ffff640 1 --2- 172.21.15.60:0/2893214911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b78148c40 0x7f4b780d2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:06.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b75d9b640 1 -- 172.21.15.60:0/2893214911 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1014725089 0 0) 0x7f4b64003520 con 0x7f4b780b7080 2024-05-09T05:01:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b75d9b640 1 -- 172.21.15.60:0/2893214911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b78148c40 msgr2=0x7f4b780d2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b75d9b640 1 --2- 172.21.15.60:0/2893214911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b78148c40 0x7f4b780d2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b75d9b640 1 -- 172.21.15.60:0/2893214911 >> v1:172.21.15.114:6789/0 conn(0x7f4b781486d0 legacy=0x7f4b780b5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b75d9b640 1 -- 172.21.15.60:0/2893214911 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b78072020 con 0x7f4b780b7080 2024-05-09T05:01:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b75d9b640 1 -- 172.21.15.60:0/2893214911 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f4b70001a70 con 0x7f4b780b7080 2024-05-09T05:01:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b6ffff640 1 --2- 172.21.15.60:0/2893214911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b78148c40 0x7f4b780d2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b75d9b640 1 -- 172.21.15.60:0/2893214911 >> v1:172.21.15.60:6789/0 conn(0x7f4b780b7080 legacy=0x7f4b78072820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:06.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b75d9b640 1 --2- 172.21.15.60:0/2893214911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b64003a80 0x7f4b64003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b75d9b640 1 --2- 172.21.15.60:0/2893214911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b64004800 0x7f4b64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b75d9b640 1 --2- 172.21.15.60:0/2893214911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b78148c40 0x7f4b640052b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b75d9b640 1 -- 172.21.15.60:0/2893214911 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4b78071810 con 0x7f4b64003a80 2024-05-09T05:01:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b75d9b640 1 -- 172.21.15.60:0/2893214911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6400bc30 con 0x7f4b64004800 2024-05-09T05:01:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b75d9b640 1 -- 172.21.15.60:0/2893214911 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4b6400be30 con 0x7f4b78148c40 2024-05-09T05:01:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b7659c640 1 --2- 172.21.15.60:0/2893214911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b64003a80 0x7f4b64003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:06.147 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b6ffff640 1 --2- 172.21.15.60:0/2893214911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b64004800 0x7f4b64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.135+0000 7f4b76d9d640 1 --2- 172.21.15.60:0/2893214911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b78148c40 0x7f4b640052b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.136+0000 7f4b7659c640 1 -- 172.21.15.60:0/2893214911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b78148c40 msgr2=0x7f4b640052b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.136+0000 7f4b7659c640 1 --2- 172.21.15.60:0/2893214911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b78148c40 0x7f4b640052b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.136+0000 7f4b7659c640 1 -- 172.21.15.60:0/2893214911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b64004800 msgr2=0x7f4b64004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.136+0000 7f4b7659c640 1 --2- 172.21.15.60:0/2893214911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b64004800 0x7f4b64004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.136+0000 7f4b7659c640 1 -- 172.21.15.60:0/2893214911 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b64003520 con 0x7f4b64003a80 2024-05-09T05:01:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.136+0000 7f4b76d9d640 1 --2- 172.21.15.60:0/2893214911 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b78148c40 0x7f4b640052b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.136+0000 7f4b7659c640 1 --2- 172.21.15.60:0/2893214911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b64003a80 0x7f4b64003ea0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4b68001730 tx=0x7f4b64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce8b4202f7381dc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.136+0000 7f4b75d9b640 1 -- 172.21.15.60:0/2893214911 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68001350 con 0x7f4b64003a80 2024-05-09T05:01:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.137+0000 7f4b75d9b640 1 -- 172.21.15.60:0/2893214911 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b680052d0 con 0x7f4b64003a80 2024-05-09T05:01:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.137+0000 7f4b75d9b640 1 -- 172.21.15.60:0/2893214911 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6802aac0 con 0x7f4b64003a80 2024-05-09T05:01:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.137+0000 7f4b7cbc1640 1 -- 172.21.15.60:0/2893214911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b64003a80 msgr2=0x7f4b64003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.137+0000 7f4b7cbc1640 1 --2- 172.21.15.60:0/2893214911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b64003a80 0x7f4b64003ea0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4b68001730 tx=0x7f4b64005830 comp rx=0 tx=0).stop 2024-05-09T05:01:06.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.137+0000 7f4b76d9d640 1 -- 172.21.15.60:0/2893214911 reap_dead start 2024-05-09T05:01:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.137+0000 7f4b7cbc1640 1 -- 172.21.15.60:0/2893214911 shutdown_connections 2024-05-09T05:01:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.137+0000 7f4b7cbc1640 1 -- 172.21.15.60:0/2893214911 >> 172.21.15.60:0/2893214911 conn(0x7f4b7806f6f0 msgr2=0x7f4b780ba5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.137+0000 7f4b7cbc1640 1 -- 172.21.15.60:0/2893214911 shutdown_connections 2024-05-09T05:01:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.137+0000 7f4b7cbc1640 1 -- 172.21.15.60:0/2893214911 wait complete. 2024-05-09T05:01:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.139+0000 7f4b7cbc1640 1 Processor -- start 2024-05-09T05:01:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.139+0000 7f4b7cbc1640 1 -- start start 2024-05-09T05:01:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.139+0000 7f4b7cbc1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b64003a80 0x7f4b7815c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.139+0000 7f4b7cbc1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b781407e0 0x7f4b781e9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.139+0000 7f4b7cbc1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b780b9d80 0x7f4b780ba200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.139+0000 7f4b7cbc1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4b780730b0 con 0x7f4b64003a80 2024-05-09T05:01:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.139+0000 7f4b7cbc1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4b78072f30 con 0x7f4b781407e0 2024-05-09T05:01:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.139+0000 7f4b7cbc1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4b78073230 con 0x7f4b780b9d80 2024-05-09T05:01:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.139+0000 7f4b7659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b64003a80 0x7f4b7815c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.139+0000 7f4b7659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b64003a80 0x7f4b7815c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56282/0 (socket says 172.21.15.60:56282) 2024-05-09T05:01:06.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.139+0000 7f4b7659c640 1 -- 172.21.15.60:0/3248553140 learned_addr learned my addr 172.21.15.60:0/3248553140 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:06.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.140+0000 7f4b6ffff640 1 --2- 172.21.15.60:0/3248553140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b781407e0 0x7f4b781e9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:06.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.140+0000 7f4b76d9d640 1 --2- 172.21.15.60:0/3248553140 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b780b9d80 0x7f4b780ba200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:06.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.140+0000 7f4b7659c640 1 -- 172.21.15.60:0/3248553140 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b780b9d80 msgr2=0x7f4b780ba200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:06.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.140+0000 7f4b7659c640 1 --2- 172.21.15.60:0/3248553140 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b780b9d80 0x7f4b780ba200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:06.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.140+0000 7f4b7659c640 1 -- 172.21.15.60:0/3248553140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b781407e0 msgr2=0x7f4b781e9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:06.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.140+0000 7f4b7659c640 1 --2- 172.21.15.60:0/3248553140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b781407e0 0x7f4b781e9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:06.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.140+0000 7f4b7659c640 1 -- 172.21.15.60:0/3248553140 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b78140c60 con 0x7f4b64003a80 2024-05-09T05:01:06.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.140+0000 7f4b7659c640 1 --2- 172.21.15.60:0/3248553140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b64003a80 0x7f4b7815c6f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4b6802a570 tx=0x7f4b68001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:06.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.140+0000 7f4b6effd640 1 -- 172.21.15.60:0/3248553140 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6803f020 con 0x7f4b64003a80 2024-05-09T05:01:06.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.140+0000 7f4b6effd640 1 -- 172.21.15.60:0/3248553140 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b68005480 con 0x7f4b64003a80 2024-05-09T05:01:06.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.140+0000 7f4b7cbc1640 1 -- 172.21.15.60:0/3248553140 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b781601f0 con 0x7f4b64003a80 2024-05-09T05:01:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.140+0000 7f4b6effd640 1 -- 172.21.15.60:0/3248553140 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68037040 con 0x7f4b64003a80 2024-05-09T05:01:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.140+0000 7f4b7cbc1640 1 -- 172.21.15.60:0/3248553140 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b781ea4a0 con 0x7f4b64003a80 2024-05-09T05:01:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.141+0000 7f4b7cbc1640 1 -- 172.21.15.60:0/3248553140 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b78051040 con 0x7f4b64003a80 2024-05-09T05:01:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.142+0000 7f4b6effd640 1 -- 172.21.15.60:0/3248553140 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4b68042020 con 0x7f4b64003a80 2024-05-09T05:01:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.142+0000 7f4b6effd640 1 --2- 172.21.15.60:0/3248553140 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4b580762d0 0x7f4b58078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.142+0000 7f4b6effd640 1 -- 172.21.15.60:0/3248553140 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6101+0+0 (secure 0 0 0) 0x7f4b680b7400 con 0x7f4b64003a80 2024-05-09T05:01:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.143+0000 7f4b6ffff640 1 --2- 172.21.15.60:0/3248553140 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4b580762d0 0x7f4b58078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.143+0000 7f4b6ffff640 1 --2- 172.21.15.60:0/3248553140 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4b580762d0 0x7f4b58078790 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4b60006fa0 tx=0x7f4b60008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.147+0000 7f4b6effd640 1 -- 172.21.15.60:0/3248553140 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4b68049020 con 0x7f4b64003a80 2024-05-09T05:01:06.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:06.378+0000 7f4b7cbc1640 1 -- 172.21.15.60:0/3248553140 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f4b78073980 con 0x7f4b64003a80 2024-05-09T05:01:07.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.346+0000 7f4b6effd640 1 -- 172.21.15.60:0/3248553140 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7f4b78140c60 con 0x7f4b64003a80 2024-05-09T05:01:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.349+0000 7f4b7cbc1640 1 -- 172.21.15.60:0/3248553140 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4b580762d0 msgr2=0x7f4b58078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.349+0000 7f4b7cbc1640 1 --2- 172.21.15.60:0/3248553140 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4b580762d0 0x7f4b58078790 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4b60006fa0 tx=0x7f4b60008040 comp rx=0 tx=0).stop 2024-05-09T05:01:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.349+0000 7f4b7cbc1640 1 -- 172.21.15.60:0/3248553140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b64003a80 msgr2=0x7f4b7815c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.349+0000 7f4b7cbc1640 1 --2- 172.21.15.60:0/3248553140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b64003a80 0x7f4b7815c6f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4b6802a570 tx=0x7f4b68001120 comp rx=0 tx=0).stop 2024-05-09T05:01:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.349+0000 7f4b7cbc1640 1 -- 172.21.15.60:0/3248553140 shutdown_connections 2024-05-09T05:01:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.349+0000 7f4b7cbc1640 1 --2- 172.21.15.60:0/3248553140 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4b580762d0 0x7f4b58078790 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.350+0000 7f4b7cbc1640 1 --2- 172.21.15.60:0/3248553140 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4b780b9d80 0x7f4b780ba200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.350+0000 7f4b7cbc1640 1 --2- 172.21.15.60:0/3248553140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b781407e0 0x7f4b781e9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.350+0000 7f4b7cbc1640 1 --2- 172.21.15.60:0/3248553140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b64003a80 0x7f4b7815c6f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.350+0000 7f4b7cbc1640 1 -- 172.21.15.60:0/3248553140 >> 172.21.15.60:0/3248553140 conn(0x7f4b7806f6f0 msgr2=0x7f4b780b8810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.350+0000 7f4b7cbc1640 1 -- 172.21.15.60:0/3248553140 shutdown_connections 2024-05-09T05:01:07.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.350+0000 7f4b7cbc1640 1 -- 172.21.15.60:0/3248553140 wait complete. 2024-05-09T05:01:07.364 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-09T05:01:07.364 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-09T05:01:07.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.477+0000 7fd6d6e7b640 1 Processor -- start 2024-05-09T05:01:07.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.477+0000 7fd6d6e7b640 1 -- start start 2024-05-09T05:01:07.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.478+0000 7fd6d6e7b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6d01482d0 0x7fd6d016a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:07.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.478+0000 7fd6d6e7b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6d00d2290 con 0x7fd6d006c370 2024-05-09T05:01:07.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.478+0000 7fd6d6e7b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6d00cc770 con 0x7fd6d0147ef0 2024-05-09T05:01:07.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.478+0000 7fd6d6e7b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd6d0149060 con 0x7fd6d01482d0 2024-05-09T05:01:07.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.478+0000 7fd6c7fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd6d006c370 0x7fd6d00d1b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51530/0 (socket says 172.21.15.60:51530) 2024-05-09T05:01:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.478+0000 7fd6c7fff640 1 -- 172.21.15.60:0/4175259650 learned_addr learned my addr 172.21.15.60:0/4175259650 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.478+0000 7fd6d5418640 1 --2- 172.21.15.60:0/4175259650 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6d01482d0 0x7fd6d016a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.478+0000 7fd6c77fe640 1 -- 172.21.15.60:0/4175259650 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4265580169 0 0) 0x7fd6d00d2290 con 0x7fd6d006c370 2024-05-09T05:01:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.478+0000 7fd6c77fe640 1 -- 172.21.15.60:0/4175259650 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6b8003520 con 0x7fd6d006c370 2024-05-09T05:01:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.478+0000 7fd6c77fe640 1 -- 172.21.15.60:0/4175259650 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2712398206 0 0) 0x7fd6d00cc770 con 0x7fd6d0147ef0 2024-05-09T05:01:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.478+0000 7fd6c77fe640 1 -- 172.21.15.60:0/4175259650 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6d00d2290 con 0x7fd6d0147ef0 2024-05-09T05:01:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.478+0000 7fd6c77fe640 1 -- 172.21.15.60:0/4175259650 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2572026875 0 0) 0x7fd6b8003520 con 0x7fd6d006c370 2024-05-09T05:01:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6c77fe640 1 -- 172.21.15.60:0/4175259650 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6d01482d0 msgr2=0x7fd6d016a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6c77fe640 1 --2- 172.21.15.60:0/4175259650 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6d01482d0 0x7fd6d016a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6c77fe640 1 -- 172.21.15.60:0/4175259650 >> v1:172.21.15.114:6789/0 conn(0x7fd6d0147ef0 legacy=0x7fd6d00d1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6c77fe640 1 -- 172.21.15.60:0/4175259650 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d00cca50 con 0x7fd6d006c370 2024-05-09T05:01:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6c77fe640 1 -- 172.21.15.60:0/4175259650 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7fd6c00016f0 con 0x7fd6d006c370 2024-05-09T05:01:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6c77fe640 1 -- 172.21.15.60:0/4175259650 >> v1:172.21.15.60:6789/0 conn(0x7fd6d006c370 legacy=0x7fd6d00d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6c77fe640 1 --2- 172.21.15.60:0/4175259650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6b8003c60 0x7fd6b8004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6c77fe640 1 --2- 172.21.15.60:0/4175259650 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6d01482d0 0x7fd6b8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6c77fe640 1 --2- 172.21.15.60:0/4175259650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6b8004d90 0x7fd6b8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6c77fe640 1 -- 172.21.15.60:0/4175259650 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b800bbf0 con 0x7fd6b8003c60 2024-05-09T05:01:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6c77fe640 1 -- 172.21.15.60:0/4175259650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b800bda0 con 0x7fd6b8004d90 2024-05-09T05:01:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6c77fe640 1 -- 172.21.15.60:0/4175259650 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd6b800bfa0 con 0x7fd6d01482d0 2024-05-09T05:01:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6d4c17640 1 --2- 172.21.15.60:0/4175259650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6b8003c60 0x7fd6b8004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6c7fff640 1 --2- 172.21.15.60:0/4175259650 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6d01482d0 0x7fd6b8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6d5418640 1 --2- 172.21.15.60:0/4175259650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6b8004d90 0x7fd6b8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6d4c17640 1 -- 172.21.15.60:0/4175259650 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6d01482d0 msgr2=0x7fd6b8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6d4c17640 1 --2- 172.21.15.60:0/4175259650 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6d01482d0 0x7fd6b8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6d4c17640 1 -- 172.21.15.60:0/4175259650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6b8004d90 msgr2=0x7fd6b8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6d4c17640 1 --2- 172.21.15.60:0/4175259650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6b8004d90 0x7fd6b8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6d4c17640 1 -- 172.21.15.60:0/4175259650 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6b8003520 con 0x7fd6b8003c60 2024-05-09T05:01:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.479+0000 7fd6d5418640 1 --2- 172.21.15.60:0/4175259650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6b8004d90 0x7fd6b8005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:07.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.480+0000 7fd6d4c17640 1 --2- 172.21.15.60:0/4175259650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6b8003c60 0x7fd6b8004060 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0018a0 tx=0x7fd6b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7cc984553265ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:07.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.480+0000 7fd6c77fe640 1 -- 172.21.15.60:0/4175259650 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc0014c0 con 0x7fd6b8003c60 2024-05-09T05:01:07.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.480+0000 7fd6c77fe640 1 -- 172.21.15.60:0/4175259650 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6cc02acf0 con 0x7fd6b8003c60 2024-05-09T05:01:07.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.480+0000 7fd6c77fe640 1 -- 172.21.15.60:0/4175259650 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc005310 con 0x7fd6b8003c60 2024-05-09T05:01:07.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.480+0000 7fd6d6e7b640 1 -- 172.21.15.60:0/4175259650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6b8003c60 msgr2=0x7fd6b8004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:07.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.480+0000 7fd6d6e7b640 1 --2- 172.21.15.60:0/4175259650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6b8003c60 0x7fd6b8004060 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0018a0 tx=0x7fd6b8005830 comp rx=0 tx=0).stop 2024-05-09T05:01:07.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.480+0000 7fd6d5418640 1 -- 172.21.15.60:0/4175259650 reap_dead start 2024-05-09T05:01:07.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.480+0000 7fd6d6e7b640 1 -- 172.21.15.60:0/4175259650 shutdown_connections 2024-05-09T05:01:07.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.480+0000 7fd6d6e7b640 1 -- 172.21.15.60:0/4175259650 >> 172.21.15.60:0/4175259650 conn(0x7fd6d00b3320 msgr2=0x7fd6d0070590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:07.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.481+0000 7fd6d6e7b640 1 -- 172.21.15.60:0/4175259650 shutdown_connections 2024-05-09T05:01:07.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.481+0000 7fd6d6e7b640 1 -- 172.21.15.60:0/4175259650 wait complete. 2024-05-09T05:01:07.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.482+0000 7fd6d6e7b640 1 Processor -- start 2024-05-09T05:01:07.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.482+0000 7fd6d6e7b640 1 -- start start 2024-05-09T05:01:07.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.483+0000 7fd6d6e7b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6b8003c60 0x7fd6d015e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:07.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.483+0000 7fd6d6e7b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6d0147130 0x7fd6d01423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:07.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.483+0000 7fd6d6e7b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6d0142900 0x7fd6d0143dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:07.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.483+0000 7fd6d6e7b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d00d2e70 con 0x7fd6d0142900 2024-05-09T05:01:07.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.483+0000 7fd6d6e7b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d00ccd30 con 0x7fd6d0147130 2024-05-09T05:01:07.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.483+0000 7fd6d6e7b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd6d01487c0 con 0x7fd6b8003c60 2024-05-09T05:01:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.483+0000 7fd6d5418640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6d0142900 0x7fd6d0143dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.483+0000 7fd6d5418640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6d0142900 0x7fd6d0143dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56304/0 (socket says 172.21.15.60:56304) 2024-05-09T05:01:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.483+0000 7fd6d5418640 1 -- 172.21.15.60:0/88753542 learned_addr learned my addr 172.21.15.60:0/88753542 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.483+0000 7fd6d4c17640 1 --2- 172.21.15.60:0/88753542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6b8003c60 0x7fd6d015e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.483+0000 7fd6c7fff640 1 --2- 172.21.15.60:0/88753542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6d0147130 0x7fd6d01423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.483+0000 7fd6d5418640 1 -- 172.21.15.60:0/88753542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6b8003c60 msgr2=0x7fd6d015e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.483+0000 7fd6d5418640 1 --2- 172.21.15.60:0/88753542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6b8003c60 0x7fd6d015e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.483+0000 7fd6d5418640 1 -- 172.21.15.60:0/88753542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6d0147130 msgr2=0x7fd6d01423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.483+0000 7fd6d5418640 1 --2- 172.21.15.60:0/88753542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6d0147130 0x7fd6d01423c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.483+0000 7fd6d5418640 1 -- 172.21.15.60:0/88753542 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d0146090 con 0x7fd6d0142900 2024-05-09T05:01:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.483+0000 7fd6c7fff640 1 --2- 172.21.15.60:0/88753542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6d0147130 0x7fd6d01423c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.483+0000 7fd6d5418640 1 --2- 172.21.15.60:0/88753542 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6d0142900 0x7fd6d0143dd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00b4f20 tx=0x7fd6c8010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.484+0000 7fd6c57fa640 1 -- 172.21.15.60:0/88753542 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8003990 con 0x7fd6d0142900 2024-05-09T05:01:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.484+0000 7fd6c57fa640 1 -- 172.21.15.60:0/88753542 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c8003b50 con 0x7fd6d0142900 2024-05-09T05:01:07.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.484+0000 7fd6d6e7b640 1 -- 172.21.15.60:0/88753542 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d0146370 con 0x7fd6d0142900 2024-05-09T05:01:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.484+0000 7fd6c57fa640 1 -- 172.21.15.60:0/88753542 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8004020 con 0x7fd6d0142900 2024-05-09T05:01:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.484+0000 7fd6d6e7b640 1 -- 172.21.15.60:0/88753542 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d0146930 con 0x7fd6d0142900 2024-05-09T05:01:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.485+0000 7fd6d6e7b640 1 -- 172.21.15.60:0/88753542 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6d0051040 con 0x7fd6d0142900 2024-05-09T05:01:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.485+0000 7fd6c57fa640 1 -- 172.21.15.60:0/88753542 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd6c8005d10 con 0x7fd6d0142900 2024-05-09T05:01:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.486+0000 7fd6c57fa640 1 --2- 172.21.15.60:0/88753542 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd6bc076300 0x7fd6bc0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.486+0000 7fd6c57fa640 1 -- 172.21.15.60:0/88753542 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7fd6c8097e10 con 0x7fd6d0142900 2024-05-09T05:01:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.486+0000 7fd6d4c17640 1 --2- 172.21.15.60:0/88753542 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd6bc076300 0x7fd6bc0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:07.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.487+0000 7fd6d4c17640 1 --2- 172.21.15.60:0/88753542 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd6bc076300 0x7fd6bc0787c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc031000 tx=0x7fd6cc036040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:07.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.491+0000 7fd6c57fa640 1 -- 172.21.15.60:0/88753542 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd6c8025020 con 0x7fd6d0142900 2024-05-09T05:01:07.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:07.720+0000 7fd6d6e7b640 1 -- 172.21.15.60:0/88753542 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fd6d0161de0 con 0x7fd6d0142900 2024-05-09T05:01:08.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.348+0000 7fd6c57fa640 1 -- 172.21.15.60:0/88753542 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) ==== 122+0+0 (secure 0 0 0) 0x7fd6c80615e0 con 0x7fd6d0142900 2024-05-09T05:01:08.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.351+0000 7fd6d6e7b640 1 -- 172.21.15.60:0/88753542 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd6bc076300 msgr2=0x7fd6bc0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:08.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.351+0000 7fd6d6e7b640 1 --2- 172.21.15.60:0/88753542 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd6bc076300 0x7fd6bc0787c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc031000 tx=0x7fd6cc036040 comp rx=0 tx=0).stop 2024-05-09T05:01:08.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.351+0000 7fd6d6e7b640 1 -- 172.21.15.60:0/88753542 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6d0142900 msgr2=0x7fd6d0143dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:08.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.351+0000 7fd6d6e7b640 1 --2- 172.21.15.60:0/88753542 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6d0142900 0x7fd6d0143dd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00b4f20 tx=0x7fd6c8010590 comp rx=0 tx=0).stop 2024-05-09T05:01:08.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.351+0000 7fd6d6e7b640 1 -- 172.21.15.60:0/88753542 shutdown_connections 2024-05-09T05:01:08.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.351+0000 7fd6d6e7b640 1 --2- 172.21.15.60:0/88753542 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd6bc076300 0x7fd6bc0787c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:08.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.351+0000 7fd6d6e7b640 1 --2- 172.21.15.60:0/88753542 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6d0142900 0x7fd6d0143dd0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:08.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.351+0000 7fd6d6e7b640 1 --2- 172.21.15.60:0/88753542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6d0147130 0x7fd6d01423c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:08.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.351+0000 7fd6d6e7b640 1 --2- 172.21.15.60:0/88753542 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6b8003c60 0x7fd6d015e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:08.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.351+0000 7fd6d6e7b640 1 -- 172.21.15.60:0/88753542 >> 172.21.15.60:0/88753542 conn(0x7fd6d00b3320 msgr2=0x7fd6d006d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:08.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.352+0000 7fd6d6e7b640 1 -- 172.21.15.60:0/88753542 shutdown_connections 2024-05-09T05:01:08.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.352+0000 7fd6d6e7b640 1 -- 172.21.15.60:0/88753542 wait complete. 2024-05-09T05:01:08.366 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-09T05:01:08.366 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:08.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.476+0000 7f5eddcda640 1 Processor -- start 2024-05-09T05:01:08.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.476+0000 7f5eddcda640 1 -- start start 2024-05-09T05:01:08.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.476+0000 7f5eddcda640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5ed80d1050 0x7f5ed80ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:08.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.476+0000 7f5eddcda640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ed8096eb0 0x7f5ed809b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:08.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.476+0000 7f5eddcda640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ed80af370 con 0x7f5ed809be90 2024-05-09T05:01:08.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.476+0000 7f5eddcda640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f5ed8096a00 con 0x7f5ed80d1050 2024-05-09T05:01:08.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.476+0000 7f5eddcda640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f5ed80d2dd0 con 0x7f5ed8096eb0 2024-05-09T05:01:08.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.477+0000 7f5ed77fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f5ed80d1050 0x7f5ed80ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:08.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.477+0000 7f5ed6ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ed8096eb0 0x7f5ed809b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:08.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.477+0000 7f5ed7fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5ed809be90 0x7f5ed80d1430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.60:52034/0 (socket says 172.21.15.60:52034) 2024-05-09T05:01:08.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.477+0000 7f5ed6ffd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ed8096eb0 0x7f5ed809b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.60:58012/0 (socket says 172.21.15.60:58012) 2024-05-09T05:01:08.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.477+0000 7f5ed7fff640 1 -- 172.21.15.60:0/1343903238 learned_addr learned my addr 172.21.15.60:0/1343903238 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:08.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.477+0000 7f5ed6ffd640 1 -- 172.21.15.60:0/1343903238 >> v2:172.21.15.114:3300/0 conn(0x7f5ed80d1050 msgr2=0x7f5ed80ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:08.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.477+0000 7f5ed6ffd640 1 --2- 172.21.15.60:0/1343903238 >> v2:172.21.15.114:3300/0 conn(0x7f5ed80d1050 0x7f5ed80ae9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:08.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.477+0000 7f5ed6ffd640 1 -- 172.21.15.60:0/1343903238 >> v1:172.21.15.114:6789/0 conn(0x7f5ed809be90 legacy=0x7f5ed80d1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:08.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.477+0000 7f5ed6ffd640 1 -- 172.21.15.60:0/1343903238 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed80d30b0 con 0x7f5ed8096eb0 2024-05-09T05:01:08.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.477+0000 7f5ed77fe640 1 --2- 172.21.15.60:0/1343903238 >> v2:172.21.15.114:3300/0 conn(0x7f5ed80d1050 0x7f5ed80ae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:01:08.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.478+0000 7f5ed6ffd640 1 --2- 172.21.15.60:0/1343903238 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ed8096eb0 0x7f5ed809b950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0097a0 tx=0x7f5ecc02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1542303883cf314a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:08.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.478+0000 7f5ed67fc640 1 -- 172.21.15.60:0/1343903238 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc037070 con 0x7f5ed8096eb0 2024-05-09T05:01:08.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.478+0000 7f5ed67fc640 1 -- 172.21.15.60:0/1343903238 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ecc032440 con 0x7f5ed8096eb0 2024-05-09T05:01:08.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.478+0000 7f5ed67fc640 1 -- 172.21.15.60:0/1343903238 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc032810 con 0x7f5ed8096eb0 2024-05-09T05:01:08.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.479+0000 7f5eddcda640 1 -- 172.21.15.60:0/1343903238 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ed8096eb0 msgr2=0x7f5ed809b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:08.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.479+0000 7f5eddcda640 1 --2- 172.21.15.60:0/1343903238 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ed8096eb0 0x7f5ed809b950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0097a0 tx=0x7f5ecc02a180 comp rx=0 tx=0).stop 2024-05-09T05:01:08.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.479+0000 7f5eddcda640 1 -- 172.21.15.60:0/1343903238 shutdown_connections 2024-05-09T05:01:08.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.479+0000 7f5eddcda640 1 --2- 172.21.15.60:0/1343903238 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ed8096eb0 0x7f5ed809b950 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:08.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.479+0000 7f5eddcda640 1 --2- 172.21.15.60:0/1343903238 >> v2:172.21.15.114:3300/0 conn(0x7f5ed80d1050 0x7f5ed80ae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:08.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.479+0000 7f5eddcda640 1 -- 172.21.15.60:0/1343903238 >> 172.21.15.60:0/1343903238 conn(0x7f5ed80993f0 msgr2=0x7f5ed8099800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:08.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.479+0000 7f5eddcda640 1 -- 172.21.15.60:0/1343903238 shutdown_connections 2024-05-09T05:01:08.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.479+0000 7f5eddcda640 1 -- 172.21.15.60:0/1343903238 wait complete. 2024-05-09T05:01:08.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.480+0000 7f5eddcda640 1 Processor -- start 2024-05-09T05:01:08.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.480+0000 7f5eddcda640 1 -- start start 2024-05-09T05:01:08.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.481+0000 7f5eddcda640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ed8096eb0 0x7f5ed81f5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:08.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.481+0000 7f5eddcda640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ed809be90 0x7f5ed81f7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:08.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.481+0000 7f5eddcda640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ed80d1050 0x7f5ed806e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:08.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.481+0000 7f5eddcda640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed80af0b0 con 0x7f5ed8096eb0 2024-05-09T05:01:08.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.481+0000 7f5eddcda640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed80aef30 con 0x7f5ed80d1050 2024-05-09T05:01:08.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.481+0000 7f5eddcda640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f5ed80d1b40 con 0x7f5ed809be90 2024-05-09T05:01:08.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.481+0000 7f5ed77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ed8096eb0 0x7f5ed81f5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:08.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.481+0000 7f5ed77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ed8096eb0 0x7f5ed81f5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56312/0 (socket says 172.21.15.60:56312) 2024-05-09T05:01:08.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.481+0000 7f5ed77fe640 1 -- 172.21.15.60:0/698080873 learned_addr learned my addr 172.21.15.60:0/698080873 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:08.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.481+0000 7f5ed7fff640 1 --2- 172.21.15.60:0/698080873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ed80d1050 0x7f5ed806e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:08.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.481+0000 7f5ed6ffd640 1 --2- 172.21.15.60:0/698080873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ed809be90 0x7f5ed81f7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:08.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.481+0000 7f5ed77fe640 1 -- 172.21.15.60:0/698080873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ed809be90 msgr2=0x7f5ed81f7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:08.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.481+0000 7f5ed77fe640 1 --2- 172.21.15.60:0/698080873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ed809be90 0x7f5ed81f7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:08.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.481+0000 7f5ed77fe640 1 -- 172.21.15.60:0/698080873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ed80d1050 msgr2=0x7f5ed806e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:08.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.481+0000 7f5ed77fe640 1 --2- 172.21.15.60:0/698080873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ed80d1050 0x7f5ed806e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:08.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.481+0000 7f5ed77fe640 1 -- 172.21.15.60:0/698080873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed806c5c0 con 0x7f5ed8096eb0 2024-05-09T05:01:08.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.482+0000 7f5ed77fe640 1 --2- 172.21.15.60:0/698080873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ed8096eb0 0x7f5ed81f5270 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8009f60 tx=0x7f5ec8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:08.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.482+0000 7f5ebffff640 1 -- 172.21.15.60:0/698080873 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec8018020 con 0x7f5ed8096eb0 2024-05-09T05:01:08.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.482+0000 7f5ebffff640 1 -- 172.21.15.60:0/698080873 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ec8014680 con 0x7f5ed8096eb0 2024-05-09T05:01:08.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.482+0000 7f5eddcda640 1 -- 172.21.15.60:0/698080873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ed806c8a0 con 0x7f5ed8096eb0 2024-05-09T05:01:08.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.482+0000 7f5ebffff640 1 -- 172.21.15.60:0/698080873 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec8014a50 con 0x7f5ed8096eb0 2024-05-09T05:01:08.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.482+0000 7f5eddcda640 1 -- 172.21.15.60:0/698080873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ed806ce60 con 0x7f5ed8096eb0 2024-05-09T05:01:08.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.483+0000 7f5eddcda640 1 -- 172.21.15.60:0/698080873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ed8075540 con 0x7f5ed8096eb0 2024-05-09T05:01:08.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.483+0000 7f5ebffff640 1 -- 172.21.15.60:0/698080873 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5ec8005d10 con 0x7f5ed8096eb0 2024-05-09T05:01:08.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.484+0000 7f5ebffff640 1 --2- 172.21.15.60:0/698080873 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5ec40762d0 0x7f5ec4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:08.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.484+0000 7f5ebffff640 1 -- 172.21.15.60:0/698080873 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7f5ec8098600 con 0x7f5ed8096eb0 2024-05-09T05:01:08.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.484+0000 7f5ed6ffd640 1 --2- 172.21.15.60:0/698080873 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5ec40762d0 0x7f5ec4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:08.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.485+0000 7f5ed6ffd640 1 --2- 172.21.15.60:0/698080873 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5ec40762d0 0x7f5ec4078790 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc034000 tx=0x7f5ecc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:08.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.488+0000 7f5ebffff640 1 -- 172.21.15.60:0/698080873 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5ec8022030 con 0x7f5ed8096eb0 2024-05-09T05:01:08.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.720+0000 7f5eddcda640 1 -- 172.21.15.60:0/698080873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ed8051040 con 0x7f5ed8096eb0 2024-05-09T05:01:08.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.720+0000 7f5ebffff640 1 -- 172.21.15.60:0/698080873 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5181 (secure 0 0 0) 0x7f5ec8010040 con 0x7f5ed8096eb0 2024-05-09T05:01:08.721 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:08.721 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4547,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2372252858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2372252858},{"type":"v1","addr":"172.21.15.114:6836","nonce":2372252858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1761282610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1761282610},{"type":"v1","addr":"172.21.15.114:6837","nonce":1761282610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4765,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/300408012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":300408012},{"type":"v1","addr":"172.21.15.60:6835","nonce":300408012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:04.698975+0000","modified":"2024-05-09T05:01:07.722167+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4893},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4893":{"gid":4893,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/399160914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":399160914},{"type":"v1","addr":"172.21.15.60:6834","nonce":399160914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4893,"qdb_cluster":[4893]},"id":2}]} 2024-05-09T05:01:08.721 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 17 2024-05-09T05:01:08.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.722+0000 7f5eddcda640 1 -- 172.21.15.60:0/698080873 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5ec40762d0 msgr2=0x7f5ec4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:08.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.722+0000 7f5eddcda640 1 --2- 172.21.15.60:0/698080873 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5ec40762d0 0x7f5ec4078790 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc034000 tx=0x7f5ecc02e000 comp rx=0 tx=0).stop 2024-05-09T05:01:08.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.722+0000 7f5eddcda640 1 -- 172.21.15.60:0/698080873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ed8096eb0 msgr2=0x7f5ed81f5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:08.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.722+0000 7f5eddcda640 1 --2- 172.21.15.60:0/698080873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ed8096eb0 0x7f5ed81f5270 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8009f60 tx=0x7f5ec8009f90 comp rx=0 tx=0).stop 2024-05-09T05:01:08.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.722+0000 7f5eddcda640 1 -- 172.21.15.60:0/698080873 shutdown_connections 2024-05-09T05:01:08.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.722+0000 7f5eddcda640 1 --2- 172.21.15.60:0/698080873 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5ec40762d0 0x7f5ec4078790 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:08.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.722+0000 7f5eddcda640 1 --2- 172.21.15.60:0/698080873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ed80d1050 0x7f5ed806e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:08.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.722+0000 7f5eddcda640 1 --2- 172.21.15.60:0/698080873 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f5ed809be90 0x7f5ed81f7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:08.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.722+0000 7f5eddcda640 1 --2- 172.21.15.60:0/698080873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ed8096eb0 0x7f5ed81f5270 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:08.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.722+0000 7f5eddcda640 1 -- 172.21.15.60:0/698080873 >> 172.21.15.60:0/698080873 conn(0x7f5ed80993f0 msgr2=0x7f5ed8098850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:08.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.722+0000 7f5eddcda640 1 -- 172.21.15.60:0/698080873 shutdown_connections 2024-05-09T05:01:08.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.722+0000 7f5eddcda640 1 -- 172.21.15.60:0/698080873 wait complete. 2024-05-09T05:01:08.734 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:01:08.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.837+0000 7f07fbf80640 1 Processor -- start 2024-05-09T05:01:08.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.838+0000 7f07fbf80640 1 -- start start 2024-05-09T05:01:08.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.838+0000 7f07fbf80640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07f415c3a0 0x7f07f4070150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:08.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.838+0000 7f07fbf80640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f07f40ac430 0x7f07f4070690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:08.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.838+0000 7f07fbf80640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07f4071b80 con 0x7f07f4097de0 2024-05-09T05:01:08.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.838+0000 7f07fbf80640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f07f4071e60 con 0x7f07f40ac430 2024-05-09T05:01:08.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.838+0000 7f07fbf80640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f07f4072140 con 0x7f07f415c3a0 2024-05-09T05:01:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07fa51d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f07f4097de0 0x7f07f4099a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33534/0 (socket says 172.21.15.60:33534) 2024-05-09T05:01:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07fa51d640 1 -- 172.21.15.60:0/4105089664 learned_addr learned my addr 172.21.15.60:0/4105089664 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f9d1c640 1 --2- 172.21.15.60:0/4105089664 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07f415c3a0 0x7f07f4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f8d1a640 1 -- 172.21.15.60:0/4105089664 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 844984943 0 0) 0x7f07f4071b80 con 0x7f07f4097de0 2024-05-09T05:01:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f951b640 1 --2- 172.21.15.60:0/4105089664 >> v2:172.21.15.114:3300/0 conn(0x7f07f40ac430 0x7f07f4070690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f8d1a640 1 -- 172.21.15.60:0/4105089664 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07e0005730 con 0x7f07f4097de0 2024-05-09T05:01:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f8d1a640 1 -- 172.21.15.60:0/4105089664 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 214881250 0 0) 0x7f07e0005730 con 0x7f07f4097de0 2024-05-09T05:01:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f8d1a640 1 -- 172.21.15.60:0/4105089664 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07f415c3a0 msgr2=0x7f07f4070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f8d1a640 1 --2- 172.21.15.60:0/4105089664 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07f415c3a0 0x7f07f4070150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f8d1a640 1 -- 172.21.15.60:0/4105089664 >> v2:172.21.15.114:3300/0 conn(0x7f07f40ac430 msgr2=0x7f07f4070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f8d1a640 1 --2- 172.21.15.60:0/4105089664 >> v2:172.21.15.114:3300/0 conn(0x7f07f40ac430 0x7f07f4070690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f8d1a640 1 -- 172.21.15.60:0/4105089664 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f4072420 con 0x7f07f4097de0 2024-05-09T05:01:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f8d1a640 1 -- 172.21.15.60:0/4105089664 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f07ec001a70 con 0x7f07f4097de0 2024-05-09T05:01:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f8d1a640 1 -- 172.21.15.60:0/4105089664 >> v1:172.21.15.60:6789/0 conn(0x7f07f4097de0 legacy=0x7f07f4099a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f8d1a640 1 --2- 172.21.15.60:0/4105089664 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07f415c3a0 0x7f07e0005c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f8d1a640 1 --2- 172.21.15.60:0/4105089664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07e00061e0 0x7f07e0006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f8d1a640 1 --2- 172.21.15.60:0/4105089664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07e0006f50 0x7f07e00073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f8d1a640 1 -- 172.21.15.60:0/4105089664 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f07f4071e60 con 0x7f07e00061e0 2024-05-09T05:01:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f8d1a640 1 -- 172.21.15.60:0/4105089664 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f07f4072140 con 0x7f07e0006f50 2024-05-09T05:01:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f8d1a640 1 -- 172.21.15.60:0/4105089664 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f07e000dc20 con 0x7f07f415c3a0 2024-05-09T05:01:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f951b640 1 --2- 172.21.15.60:0/4105089664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07e00061e0 0x7f07e0006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07f9d1c640 1 --2- 172.21.15.60:0/4105089664 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07f415c3a0 0x7f07e0005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.839+0000 7f07fa51d640 1 --2- 172.21.15.60:0/4105089664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07e0006f50 0x7f07e00073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.840+0000 7f07f951b640 1 -- 172.21.15.60:0/4105089664 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07f415c3a0 msgr2=0x7f07e0005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.840+0000 7f07f951b640 1 --2- 172.21.15.60:0/4105089664 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07f415c3a0 0x7f07e0005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.840+0000 7f07f951b640 1 -- 172.21.15.60:0/4105089664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07e0006f50 msgr2=0x7f07e00073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.840+0000 7f07f951b640 1 --2- 172.21.15.60:0/4105089664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07e0006f50 0x7f07e00073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:08.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.840+0000 7f07f951b640 1 -- 172.21.15.60:0/4105089664 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e0005730 con 0x7f07e00061e0 2024-05-09T05:01:08.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.840+0000 7f07f951b640 1 --2- 172.21.15.60:0/4105089664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07e00061e0 0x7f07e0006a10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f07e4009ac0 tx=0x7f07e0007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f53b0c9a9a8881d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:08.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.840+0000 7f07f8d1a640 1 -- 172.21.15.60:0/4105089664 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e4036070 con 0x7f07e00061e0 2024-05-09T05:01:08.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.840+0000 7f07f8d1a640 1 -- 172.21.15.60:0/4105089664 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07e4031400 con 0x7f07e00061e0 2024-05-09T05:01:08.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.840+0000 7f07f8d1a640 1 -- 172.21.15.60:0/4105089664 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e4009690 con 0x7f07e00061e0 2024-05-09T05:01:08.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.841+0000 7f07fbf80640 1 -- 172.21.15.60:0/4105089664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07e00061e0 msgr2=0x7f07e0006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.841+0000 7f07fbf80640 1 --2- 172.21.15.60:0/4105089664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07e00061e0 0x7f07e0006a10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f07e4009ac0 tx=0x7f07e0007a40 comp rx=0 tx=0).stop 2024-05-09T05:01:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.841+0000 7f07fa51d640 1 -- 172.21.15.60:0/4105089664 reap_dead start 2024-05-09T05:01:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.841+0000 7f07fbf80640 1 -- 172.21.15.60:0/4105089664 shutdown_connections 2024-05-09T05:01:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.841+0000 7f07fbf80640 1 -- 172.21.15.60:0/4105089664 >> 172.21.15.60:0/4105089664 conn(0x7f07f40972f0 msgr2=0x7f07f40976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.841+0000 7f07fbf80640 1 -- 172.21.15.60:0/4105089664 shutdown_connections 2024-05-09T05:01:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.841+0000 7f07fbf80640 1 -- 172.21.15.60:0/4105089664 wait complete. 2024-05-09T05:01:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.842+0000 7f07fbf80640 1 Processor -- start 2024-05-09T05:01:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.842+0000 7f07fbf80640 1 -- start start 2024-05-09T05:01:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.843+0000 7f07fbf80640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07e00061e0 0x7f07f4154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.843+0000 7f07fbf80640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07f40ab310 0x7f07f414abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.843+0000 7f07fbf80640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07f40aa3a0 0x7f07f40aa830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.843+0000 7f07fbf80640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f07f4071810 con 0x7f07f40aa3a0 2024-05-09T05:01:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.843+0000 7f07fbf80640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f07f415c780 con 0x7f07e00061e0 2024-05-09T05:01:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.843+0000 7f07fbf80640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f07f4073c30 con 0x7f07f40ab310 2024-05-09T05:01:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.843+0000 7f07fa51d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07f40aa3a0 0x7f07f40aa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.843+0000 7f07fa51d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07f40aa3a0 0x7f07f40aa830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59776/0 (socket says 172.21.15.60:59776) 2024-05-09T05:01:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.843+0000 7f07fa51d640 1 -- 172.21.15.60:0/2445891171 learned_addr learned my addr 172.21.15.60:0/2445891171 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.843+0000 7f07fa51d640 1 -- 172.21.15.60:0/2445891171 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07f40ab310 msgr2=0x7f07f414abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.844+0000 7f07f951b640 1 --2- 172.21.15.60:0/2445891171 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07f40ab310 0x7f07f414abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.844+0000 7f07f9d1c640 1 --2- 172.21.15.60:0/2445891171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07e00061e0 0x7f07f4154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.844+0000 7f07fa51d640 1 --2- 172.21.15.60:0/2445891171 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07f40ab310 0x7f07f414abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.844+0000 7f07fa51d640 1 -- 172.21.15.60:0/2445891171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07e00061e0 msgr2=0x7f07f4154ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.844+0000 7f07fa51d640 1 --2- 172.21.15.60:0/2445891171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07e00061e0 0x7f07f4154ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.844+0000 7f07fa51d640 1 -- 172.21.15.60:0/2445891171 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f40aec30 con 0x7f07f40aa3a0 2024-05-09T05:01:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.844+0000 7f07fa51d640 1 --2- 172.21.15.60:0/2445891171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07f40aa3a0 0x7f07f40aa830 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f07ec002230 tx=0x7f07ec00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.844+0000 7f07ea7fc640 1 -- 172.21.15.60:0/2445891171 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec00eb90 con 0x7f07f40aa3a0 2024-05-09T05:01:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.844+0000 7f07ea7fc640 1 -- 172.21.15.60:0/2445891171 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07ec00ed30 con 0x7f07f40aa3a0 2024-05-09T05:01:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.844+0000 7f07fbf80640 1 -- 172.21.15.60:0/2445891171 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07f40aef10 con 0x7f07f40aa3a0 2024-05-09T05:01:08.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.844+0000 7f07ea7fc640 1 -- 172.21.15.60:0/2445891171 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec014670 con 0x7f07f40aa3a0 2024-05-09T05:01:08.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.844+0000 7f07fbf80640 1 -- 172.21.15.60:0/2445891171 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07f40af4d0 con 0x7f07f40aa3a0 2024-05-09T05:01:08.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.845+0000 7f07fbf80640 1 -- 172.21.15.60:0/2445891171 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07f4051040 con 0x7f07f40aa3a0 2024-05-09T05:01:08.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.845+0000 7f07ea7fc640 1 -- 172.21.15.60:0/2445891171 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f07ec014810 con 0x7f07f40aa3a0 2024-05-09T05:01:08.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.846+0000 7f07ea7fc640 1 --2- 172.21.15.60:0/2445891171 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f07d8076300 0x7f07d80787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:08.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.846+0000 7f07ea7fc640 1 -- 172.21.15.60:0/2445891171 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7f07ec0975a0 con 0x7f07f40aa3a0 2024-05-09T05:01:08.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.846+0000 7f07f9d1c640 1 --2- 172.21.15.60:0/2445891171 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f07d8076300 0x7f07d80787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:08.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.847+0000 7f07f9d1c640 1 --2- 172.21.15.60:0/2445891171 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f07d8076300 0x7f07d80787c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f07f415b7b0 tx=0x7f07f0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:08.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:08.850+0000 7f07ea7fc640 1 -- 172.21.15.60:0/2445891171 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f07ec060d70 con 0x7f07f40aa3a0 2024-05-09T05:01:09.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.042+0000 7f07fbf80640 1 -- 172.21.15.60:0/2445891171 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f07f4159570 con 0x7f07f40aa3a0 2024-05-09T05:01:09.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.043+0000 7f07ea7fc640 1 -- 172.21.15.60:0/2445891171 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+16918 (secure 0 0 0) 0x7f07ec060670 con 0x7f07f40aa3a0 2024-05-09T05:01:09.044 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:09.044 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":36,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:01:06.338414+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:01:02.773384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-05-09T05:01:03.658182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:01:09.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.045+0000 7f07fbf80640 1 -- 172.21.15.60:0/2445891171 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f07d8076300 msgr2=0x7f07d80787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:09.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.045+0000 7f07fbf80640 1 --2- 172.21.15.60:0/2445891171 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f07d8076300 0x7f07d80787c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f07f415b7b0 tx=0x7f07f0008040 comp rx=0 tx=0).stop 2024-05-09T05:01:09.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.045+0000 7f07fbf80640 1 -- 172.21.15.60:0/2445891171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07f40aa3a0 msgr2=0x7f07f40aa830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:09.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.045+0000 7f07fbf80640 1 --2- 172.21.15.60:0/2445891171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07f40aa3a0 0x7f07f40aa830 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f07ec002230 tx=0x7f07ec00e530 comp rx=0 tx=0).stop 2024-05-09T05:01:09.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.045+0000 7f07fbf80640 1 -- 172.21.15.60:0/2445891171 shutdown_connections 2024-05-09T05:01:09.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.045+0000 7f07fbf80640 1 --2- 172.21.15.60:0/2445891171 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f07d8076300 0x7f07d80787c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.045+0000 7f07fbf80640 1 --2- 172.21.15.60:0/2445891171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07f40aa3a0 0x7f07f40aa830 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.045+0000 7f07fbf80640 1 --2- 172.21.15.60:0/2445891171 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f07f40ab310 0x7f07f414abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.045+0000 7f07fbf80640 1 --2- 172.21.15.60:0/2445891171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f07e00061e0 0x7f07f4154ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.045+0000 7f07fbf80640 1 -- 172.21.15.60:0/2445891171 >> 172.21.15.60:0/2445891171 conn(0x7f07f40972f0 msgr2=0x7f07f4072c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:09.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.045+0000 7f07fbf80640 1 -- 172.21.15.60:0/2445891171 shutdown_connections 2024-05-09T05:01:09.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.045+0000 7f07fbf80640 1 -- 172.21.15.60:0/2445891171 wait complete. 2024-05-09T05:01:09.057 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-09T05:01:09.057 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:01:09.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.160+0000 7f9bed2e1640 1 Processor -- start 2024-05-09T05:01:09.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.160+0000 7f9bed2e1640 1 -- start start 2024-05-09T05:01:09.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.161+0000 7f9bed2e1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9be8074130 0x7f9be806faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:09.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.161+0000 7f9bed2e1640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9be815dea0 con 0x7f9be8074750 2024-05-09T05:01:09.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.161+0000 7f9bed2e1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9be806ea00 con 0x7f9be815d3b0 2024-05-09T05:01:09.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.161+0000 7f9bed2e1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9be806e7b0 con 0x7f9be8074130 2024-05-09T05:01:09.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.161+0000 7f9bdffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9be8074750 0x7f9be8073a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33536/0 (socket says 172.21.15.60:33536) 2024-05-09T05:01:09.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.161+0000 7f9bdffff640 1 -- 172.21.15.60:0/2304020868 learned_addr learned my addr 172.21.15.60:0/2304020868 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:09.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.161+0000 7f9be77fe640 1 --2- 172.21.15.60:0/2304020868 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9be8074130 0x7f9be806faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:09.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.161+0000 7f9be67fc640 1 -- 172.21.15.60:0/2304020868 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3971285743 0 0) 0x7f9be815dea0 con 0x7f9be8074750 2024-05-09T05:01:09.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.161+0000 7f9be67fc640 1 -- 172.21.15.60:0/2304020868 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bd4003520 con 0x7f9be8074750 2024-05-09T05:01:09.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.161+0000 7f9be77fe640 1 -- 172.21.15.60:0/2304020868 >> v1:172.21.15.114:6789/0 conn(0x7f9be815d3b0 legacy=0x7f9be815d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:09.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.161+0000 7f9be67fc640 1 -- 172.21.15.60:0/2304020868 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2562036105 0 0) 0x7f9bd4003520 con 0x7f9be8074750 2024-05-09T05:01:09.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.161+0000 7f9be77fe640 1 -- 172.21.15.60:0/2304020868 >> v1:172.21.15.60:6789/0 conn(0x7f9be8074750 legacy=0x7f9be8073a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:09.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.161+0000 7f9be77fe640 1 -- 172.21.15.60:0/2304020868 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9be806ece0 con 0x7f9be8074130 2024-05-09T05:01:09.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.162+0000 7f9be77fe640 1 --2- 172.21.15.60:0/2304020868 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9be8074130 0x7f9be806faa0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9be806e930 tx=0x7f9be002a4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d054e713aacd2c52 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:09.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.162+0000 7f9be67fc640 1 -- 172.21.15.60:0/2304020868 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be0035020 con 0x7f9be8074130 2024-05-09T05:01:09.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.162+0000 7f9be67fc640 1 -- 172.21.15.60:0/2304020868 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9be0007540 con 0x7f9be8074130 2024-05-09T05:01:09.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.162+0000 7f9be67fc640 1 -- 172.21.15.60:0/2304020868 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd00016f0 con 0x7f9be8074130 2024-05-09T05:01:09.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.163+0000 7f9bed2e1640 1 -- 172.21.15.60:0/2304020868 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9be8074130 msgr2=0x7f9be806faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:09.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.163+0000 7f9bed2e1640 1 --2- 172.21.15.60:0/2304020868 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9be8074130 0x7f9be806faa0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9be806e930 tx=0x7f9be002a4e0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.163+0000 7f9bed2e1640 1 -- 172.21.15.60:0/2304020868 shutdown_connections 2024-05-09T05:01:09.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.163+0000 7f9bed2e1640 1 --2- 172.21.15.60:0/2304020868 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9be8074130 0x7f9be806faa0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.163+0000 7f9bed2e1640 1 -- 172.21.15.60:0/2304020868 >> 172.21.15.60:0/2304020868 conn(0x7f9be80972f0 msgr2=0x7f9be80976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:09.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.163+0000 7f9bed2e1640 1 -- 172.21.15.60:0/2304020868 shutdown_connections 2024-05-09T05:01:09.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.163+0000 7f9bed2e1640 1 -- 172.21.15.60:0/2304020868 wait complete. 2024-05-09T05:01:09.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.164+0000 7f9bed2e1640 1 Processor -- start 2024-05-09T05:01:09.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.164+0000 7f9bed2e1640 1 -- start start 2024-05-09T05:01:09.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.165+0000 7f9bed2e1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be8074130 0x7f9be815bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:09.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.165+0000 7f9bed2e1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9be8074750 0x7f9be815c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:09.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.165+0000 7f9bed2e1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9be815d3b0 0x7f9be80ac980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:09.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.165+0000 7f9bed2e1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9be806f2a0 con 0x7f9be8074750 2024-05-09T05:01:09.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.165+0000 7f9bed2e1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9be806f610 con 0x7f9be8074130 2024-05-09T05:01:09.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.165+0000 7f9bed2e1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9be80b4540 con 0x7f9be815d3b0 2024-05-09T05:01:09.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.165+0000 7f9bdffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9be8074750 0x7f9be815c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:09.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.165+0000 7f9bdffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9be8074750 0x7f9be815c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59780/0 (socket says 172.21.15.60:59780) 2024-05-09T05:01:09.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.165+0000 7f9bdffff640 1 -- 172.21.15.60:0/2238545882 learned_addr learned my addr 172.21.15.60:0/2238545882 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:09.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.165+0000 7f9be77fe640 1 --2- 172.21.15.60:0/2238545882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9be815d3b0 0x7f9be80ac980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:09.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.165+0000 7f9be6ffd640 1 --2- 172.21.15.60:0/2238545882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be8074130 0x7f9be815bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:09.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.166+0000 7f9bdffff640 1 -- 172.21.15.60:0/2238545882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9be815d3b0 msgr2=0x7f9be80ac980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:09.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.166+0000 7f9bdffff640 1 --2- 172.21.15.60:0/2238545882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9be815d3b0 0x7f9be80ac980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.166+0000 7f9bdffff640 1 -- 172.21.15.60:0/2238545882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be8074130 msgr2=0x7f9be815bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:09.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.166+0000 7f9bdffff640 1 --2- 172.21.15.60:0/2238545882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be8074130 0x7f9be815bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.166+0000 7f9bdffff640 1 -- 172.21.15.60:0/2238545882 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9be80ad700 con 0x7f9be8074750 2024-05-09T05:01:09.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.166+0000 7f9be6ffd640 1 --2- 172.21.15.60:0/2238545882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be8074130 0x7f9be815bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:09.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.166+0000 7f9bdffff640 1 --2- 172.21.15.60:0/2238545882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9be8074750 0x7f9be815c140 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9bd000c730 tx=0x7f9bd000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:09.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.166+0000 7f9bdf7fe640 1 -- 172.21.15.60:0/2238545882 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0017070 con 0x7f9be8074750 2024-05-09T05:01:09.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.166+0000 7f9bdf7fe640 1 -- 172.21.15.60:0/2238545882 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bd00072d0 con 0x7f9be8074750 2024-05-09T05:01:09.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.166+0000 7f9bed2e1640 1 -- 172.21.15.60:0/2238545882 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9be8151010 con 0x7f9be8074750 2024-05-09T05:01:09.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.166+0000 7f9bdf7fe640 1 -- 172.21.15.60:0/2238545882 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd00047e0 con 0x7f9be8074750 2024-05-09T05:01:09.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.166+0000 7f9bed2e1640 1 -- 172.21.15.60:0/2238545882 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9be81515d0 con 0x7f9be8074750 2024-05-09T05:01:09.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.167+0000 7f9bed2e1640 1 -- 172.21.15.60:0/2238545882 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9be81517d0 con 0x7f9be8074750 2024-05-09T05:01:09.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.167+0000 7f9bdf7fe640 1 -- 172.21.15.60:0/2238545882 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9bd001a070 con 0x7f9be8074750 2024-05-09T05:01:09.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.168+0000 7f9bdf7fe640 1 --2- 172.21.15.60:0/2238545882 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9bc80762d0 0x7f9bc8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:09.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.168+0000 7f9bdf7fe640 1 -- 172.21.15.60:0/2238545882 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7f9bd0096b90 con 0x7f9be8074750 2024-05-09T05:01:09.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.168+0000 7f9be6ffd640 1 --2- 172.21.15.60:0/2238545882 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9bc80762d0 0x7f9bc8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:09.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.169+0000 7f9be6ffd640 1 --2- 172.21.15.60:0/2238545882 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9bc80762d0 0x7f9bc8078790 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8007fd0 tx=0x7f9bd8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:09.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.172+0000 7f9bdf7fe640 1 -- 172.21.15.60:0/2238545882 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9be81517d0 con 0x7f9be8074750 2024-05-09T05:01:09.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.362+0000 7f9bed2e1640 1 -- 172.21.15.60:0/2238545882 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9be80ad9e0 con 0x7f9bc80762d0 2024-05-09T05:01:09.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.369+0000 7f9bdf7fe640 1 -- 172.21.15.60:0/2238545882 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251944 (secure 0 0 0) 0x7f9bd8001ce0 con 0x7f9bc80762d0 2024-05-09T05:01:09.372 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:09.373 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-05-09T05:01:09.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.372+0000 7f9bed2e1640 1 -- 172.21.15.60:0/2238545882 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9bc80762d0 msgr2=0x7f9bc8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:09.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.372+0000 7f9bed2e1640 1 --2- 172.21.15.60:0/2238545882 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9bc80762d0 0x7f9bc8078790 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8007fd0 tx=0x7f9bd8009040 comp rx=0 tx=0).stop 2024-05-09T05:01:09.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.372+0000 7f9bed2e1640 1 -- 172.21.15.60:0/2238545882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9be8074750 msgr2=0x7f9be815c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:09.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.372+0000 7f9bed2e1640 1 --2- 172.21.15.60:0/2238545882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9be8074750 0x7f9be815c140 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9bd000c730 tx=0x7f9bd000cbf0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.372+0000 7f9bed2e1640 1 -- 172.21.15.60:0/2238545882 shutdown_connections 2024-05-09T05:01:09.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.372+0000 7f9bed2e1640 1 --2- 172.21.15.60:0/2238545882 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9bc80762d0 0x7f9bc8078790 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.372+0000 7f9bed2e1640 1 --2- 172.21.15.60:0/2238545882 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9be815d3b0 0x7f9be80ac980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.372+0000 7f9bed2e1640 1 --2- 172.21.15.60:0/2238545882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9be8074750 0x7f9be815c140 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.372+0000 7f9bed2e1640 1 --2- 172.21.15.60:0/2238545882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be8074130 0x7f9be815bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.372+0000 7f9bed2e1640 1 -- 172.21.15.60:0/2238545882 >> 172.21.15.60:0/2238545882 conn(0x7f9be80972f0 msgr2=0x7f9be806faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:09.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.372+0000 7f9bed2e1640 1 -- 172.21.15.60:0/2238545882 shutdown_connections 2024-05-09T05:01:09.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.372+0000 7f9bed2e1640 1 -- 172.21.15.60:0/2238545882 wait complete. 2024-05-09T05:01:09.383 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-05-09T05:01:08.923907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237216,"kb_used_data":4396,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737504,"statfs":{"total":773094113280,"available":772851204096,"internally_reserved":0,"allocated":4501504,"data_stored":2554584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.605477"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344743+0000","last_change":"2024-05-09T05:01:05.342734+0000","last_active":"2024-05-09T05:01:06.344743+0000","last_peered":"2024-05-09T05:01:06.344743+0000","last_clean":"2024-05-09T05:01:06.344743+0000","last_became_active":"2024-05-09T05:01:05.342459+0000","last_became_peered":"2024-05-09T05:01:05.342459+0000","last_unstale":"2024-05-09T05:01:06.344743+0000","last_undegraded":"2024-05-09T05:01:06.344743+0000","last_fullsized":"2024-05-09T05:01:06.344743+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:18:04.101345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344653+0000","last_change":"2024-05-09T05:01:05.349559+0000","last_active":"2024-05-09T05:01:06.344653+0000","last_peered":"2024-05-09T05:01:06.344653+0000","last_clean":"2024-05-09T05:01:06.344653+0000","last_became_active":"2024-05-09T05:01:05.349207+0000","last_became_peered":"2024-05-09T05:01:05.349207+0000","last_unstale":"2024-05-09T05:01:06.344653+0000","last_undegraded":"2024-05-09T05:01:06.344653+0000","last_fullsized":"2024-05-09T05:01:06.344653+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:51:54.638779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.552957+0000","last_change":"2024-05-09T05:01:05.348781+0000","last_active":"2024-05-09T05:01:06.552957+0000","last_peered":"2024-05-09T05:01:06.552957+0000","last_clean":"2024-05-09T05:01:06.552957+0000","last_became_active":"2024-05-09T05:01:05.348507+0000","last_became_peered":"2024-05-09T05:01:05.348507+0000","last_unstale":"2024-05-09T05:01:06.552957+0000","last_undegraded":"2024-05-09T05:01:06.552957+0000","last_fullsized":"2024-05-09T05:01:06.552957+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:54:27.673057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.347746+0000","last_change":"2024-05-09T05:01:05.356233+0000","last_active":"2024-05-09T05:01:06.347746+0000","last_peered":"2024-05-09T05:01:06.347746+0000","last_clean":"2024-05-09T05:01:06.347746+0000","last_became_active":"2024-05-09T05:01:05.355862+0000","last_became_peered":"2024-05-09T05:01:05.355862+0000","last_unstale":"2024-05-09T05:01:06.347746+0000","last_undegraded":"2024-05-09T05:01:06.347746+0000","last_fullsized":"2024-05-09T05:01:06.347746+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:59:59.081918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.347626+0000","last_change":"2024-05-09T05:01:05.355637+0000","last_active":"2024-05-09T05:01:06.347626+0000","last_peered":"2024-05-09T05:01:06.347626+0000","last_clean":"2024-05-09T05:01:06.347626+0000","last_became_active":"2024-05-09T05:01:05.353042+0000","last_became_peered":"2024-05-09T05:01:05.353042+0000","last_unstale":"2024-05-09T05:01:06.347626+0000","last_undegraded":"2024-05-09T05:01:06.347626+0000","last_fullsized":"2024-05-09T05:01:06.347626+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:35.818258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.390033+0000","last_change":"2024-05-09T05:01:05.359894+0000","last_active":"2024-05-09T05:01:06.390033+0000","last_peered":"2024-05-09T05:01:06.390033+0000","last_clean":"2024-05-09T05:01:06.390033+0000","last_became_active":"2024-05-09T05:01:05.359626+0000","last_became_peered":"2024-05-09T05:01:05.359626+0000","last_unstale":"2024-05-09T05:01:06.390033+0000","last_undegraded":"2024-05-09T05:01:06.390033+0000","last_fullsized":"2024-05-09T05:01:06.390033+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:36:28.104239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.349627+0000","last_change":"2024-05-09T05:01:05.359081+0000","last_active":"2024-05-09T05:01:06.349627+0000","last_peered":"2024-05-09T05:01:06.349627+0000","last_clean":"2024-05-09T05:01:06.349627+0000","last_became_active":"2024-05-09T05:01:05.358508+0000","last_became_peered":"2024-05-09T05:01:05.358508+0000","last_unstale":"2024-05-09T05:01:06.349627+0000","last_undegraded":"2024-05-09T05:01:06.349627+0000","last_fullsized":"2024-05-09T05:01:06.349627+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:18:38.102212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344299+0000","last_change":"2024-05-09T05:01:05.344943+0000","last_active":"2024-05-09T05:01:06.344299+0000","last_peered":"2024-05-09T05:01:06.344299+0000","last_clean":"2024-05-09T05:01:06.344299+0000","last_became_active":"2024-05-09T05:01:05.344076+0000","last_became_peered":"2024-05-09T05:01:05.344076+0000","last_unstale":"2024-05-09T05:01:06.344299+0000","last_undegraded":"2024-05-09T05:01:06.344299+0000","last_fullsized":"2024-05-09T05:01:06.344299+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:04.027664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553090+0000","last_change":"2024-05-09T05:01:05.343323+0000","last_active":"2024-05-09T05:01:06.553090+0000","last_peered":"2024-05-09T05:01:06.553090+0000","last_clean":"2024-05-09T05:01:06.553090+0000","last_became_active":"2024-05-09T05:01:05.343026+0000","last_became_peered":"2024-05-09T05:01:05.343026+0000","last_unstale":"2024-05-09T05:01:06.553090+0000","last_undegraded":"2024-05-09T05:01:06.553090+0000","last_fullsized":"2024-05-09T05:01:06.553090+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:07:00.101790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.350027+0000","last_change":"2024-05-09T05:01:05.347643+0000","last_active":"2024-05-09T05:01:06.350027+0000","last_peered":"2024-05-09T05:01:06.350027+0000","last_clean":"2024-05-09T05:01:06.350027+0000","last_became_active":"2024-05-09T05:01:05.347205+0000","last_became_peered":"2024-05-09T05:01:05.347205+0000","last_unstale":"2024-05-09T05:01:06.350027+0000","last_undegraded":"2024-05-09T05:01:06.350027+0000","last_fullsized":"2024-05-09T05:01:06.350027+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:18:04.101345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.350029+0000","last_change":"2024-05-09T05:01:05.354158+0000","last_active":"2024-05-09T05:01:06.350029+0000","last_peered":"2024-05-09T05:01:06.350029+0000","last_clean":"2024-05-09T05:01:06.350029+0000","last_became_active":"2024-05-09T05:01:05.353169+0000","last_became_peered":"2024-05-09T05:01:05.353169+0000","last_unstale":"2024-05-09T05:01:06.350029+0000","last_undegraded":"2024-05-09T05:01:06.350029+0000","last_fullsized":"2024-05-09T05:01:06.350029+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:42:57.749068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.552937+0000","last_change":"2024-05-09T05:01:05.354354+0000","last_active":"2024-05-09T05:01:06.552937+0000","last_peered":"2024-05-09T05:01:06.552937+0000","last_clean":"2024-05-09T05:01:06.552937+0000","last_became_active":"2024-05-09T05:01:05.354075+0000","last_became_peered":"2024-05-09T05:01:05.354075+0000","last_unstale":"2024-05-09T05:01:06.552937+0000","last_undegraded":"2024-05-09T05:01:06.552937+0000","last_fullsized":"2024-05-09T05:01:06.552937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:39:57.651574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.349732+0000","last_change":"2024-05-09T05:01:05.346339+0000","last_active":"2024-05-09T05:01:06.349732+0000","last_peered":"2024-05-09T05:01:06.349732+0000","last_clean":"2024-05-09T05:01:06.349732+0000","last_became_active":"2024-05-09T05:01:05.345971+0000","last_became_peered":"2024-05-09T05:01:05.345971+0000","last_unstale":"2024-05-09T05:01:06.349732+0000","last_undegraded":"2024-05-09T05:01:06.349732+0000","last_fullsized":"2024-05-09T05:01:06.349732+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:22:03.591207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553527+0000","last_change":"2024-05-09T05:01:05.352479+0000","last_active":"2024-05-09T05:01:06.553527+0000","last_peered":"2024-05-09T05:01:06.553527+0000","last_clean":"2024-05-09T05:01:06.553527+0000","last_became_active":"2024-05-09T05:01:05.352215+0000","last_became_peered":"2024-05-09T05:01:05.352215+0000","last_unstale":"2024-05-09T05:01:06.553527+0000","last_undegraded":"2024-05-09T05:01:06.553527+0000","last_fullsized":"2024-05-09T05:01:06.553527+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:20.893689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.349681+0000","last_change":"2024-05-09T05:01:05.344664+0000","last_active":"2024-05-09T05:01:06.349681+0000","last_peered":"2024-05-09T05:01:06.349681+0000","last_clean":"2024-05-09T05:01:06.349681+0000","last_became_active":"2024-05-09T05:01:05.344191+0000","last_became_peered":"2024-05-09T05:01:05.344191+0000","last_unstale":"2024-05-09T05:01:06.349681+0000","last_undegraded":"2024-05-09T05:01:06.349681+0000","last_fullsized":"2024-05-09T05:01:06.349681+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:06:27.722335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.345788+0000","last_change":"2024-05-09T05:01:05.351017+0000","last_active":"2024-05-09T05:01:06.345788+0000","last_peered":"2024-05-09T05:01:06.345788+0000","last_clean":"2024-05-09T05:01:06.345788+0000","last_became_active":"2024-05-09T05:01:05.350699+0000","last_became_peered":"2024-05-09T05:01:05.350699+0000","last_unstale":"2024-05-09T05:01:06.345788+0000","last_undegraded":"2024-05-09T05:01:06.345788+0000","last_fullsized":"2024-05-09T05:01:06.345788+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:52.250650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346518+0000","last_change":"2024-05-09T05:01:05.351131+0000","last_active":"2024-05-09T05:01:06.346518+0000","last_peered":"2024-05-09T05:01:06.346518+0000","last_clean":"2024-05-09T05:01:06.346518+0000","last_became_active":"2024-05-09T05:01:05.350831+0000","last_became_peered":"2024-05-09T05:01:05.350831+0000","last_unstale":"2024-05-09T05:01:06.346518+0000","last_undegraded":"2024-05-09T05:01:06.346518+0000","last_fullsized":"2024-05-09T05:01:06.346518+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:43:59.088798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553530+0000","last_change":"2024-05-09T05:01:05.353968+0000","last_active":"2024-05-09T05:01:06.553530+0000","last_peered":"2024-05-09T05:01:06.553530+0000","last_clean":"2024-05-09T05:01:06.553530+0000","last_became_active":"2024-05-09T05:01:05.353690+0000","last_became_peered":"2024-05-09T05:01:05.353690+0000","last_unstale":"2024-05-09T05:01:06.553530+0000","last_undegraded":"2024-05-09T05:01:06.553530+0000","last_fullsized":"2024-05-09T05:01:06.553530+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:11.997036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.347847+0000","last_change":"2024-05-09T05:01:05.349945+0000","last_active":"2024-05-09T05:01:06.347847+0000","last_peered":"2024-05-09T05:01:06.347847+0000","last_clean":"2024-05-09T05:01:06.347847+0000","last_became_active":"2024-05-09T05:01:05.349644+0000","last_became_peered":"2024-05-09T05:01:05.349644+0000","last_unstale":"2024-05-09T05:01:06.347847+0000","last_undegraded":"2024-05-09T05:01:06.347847+0000","last_fullsized":"2024-05-09T05:01:06.347847+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:29.806811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553710+0000","last_change":"2024-05-09T05:01:05.353538+0000","last_active":"2024-05-09T05:01:06.553710+0000","last_peered":"2024-05-09T05:01:06.553710+0000","last_clean":"2024-05-09T05:01:06.553710+0000","last_became_active":"2024-05-09T05:01:05.353216+0000","last_became_peered":"2024-05-09T05:01:05.353216+0000","last_unstale":"2024-05-09T05:01:06.553710+0000","last_undegraded":"2024-05-09T05:01:06.553710+0000","last_fullsized":"2024-05-09T05:01:06.553710+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:38.953101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.349803+0000","last_change":"2024-05-09T05:01:05.342529+0000","last_active":"2024-05-09T05:01:06.349803+0000","last_peered":"2024-05-09T05:01:06.349803+0000","last_clean":"2024-05-09T05:01:06.349803+0000","last_became_active":"2024-05-09T05:01:05.342230+0000","last_became_peered":"2024-05-09T05:01:05.342230+0000","last_unstale":"2024-05-09T05:01:06.349803+0000","last_undegraded":"2024-05-09T05:01:06.349803+0000","last_fullsized":"2024-05-09T05:01:06.349803+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:51:54.638779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553701+0000","last_change":"2024-05-09T05:01:05.353147+0000","last_active":"2024-05-09T05:01:06.553701+0000","last_peered":"2024-05-09T05:01:06.553701+0000","last_clean":"2024-05-09T05:01:06.553701+0000","last_became_active":"2024-05-09T05:01:05.352845+0000","last_became_peered":"2024-05-09T05:01:05.352845+0000","last_unstale":"2024-05-09T05:01:06.553701+0000","last_undegraded":"2024-05-09T05:01:06.553701+0000","last_fullsized":"2024-05-09T05:01:06.553701+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:38.953101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.345816+0000","last_change":"2024-05-09T05:01:05.353291+0000","last_active":"2024-05-09T05:01:06.345816+0000","last_peered":"2024-05-09T05:01:06.345816+0000","last_clean":"2024-05-09T05:01:06.345816+0000","last_became_active":"2024-05-09T05:01:05.353024+0000","last_became_peered":"2024-05-09T05:01:05.353024+0000","last_unstale":"2024-05-09T05:01:06.345816+0000","last_undegraded":"2024-05-09T05:01:06.345816+0000","last_fullsized":"2024-05-09T05:01:06.345816+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:07:00.101790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.347489+0000","last_change":"2024-05-09T05:01:05.353091+0000","last_active":"2024-05-09T05:01:06.347489+0000","last_peered":"2024-05-09T05:01:06.347489+0000","last_clean":"2024-05-09T05:01:06.347489+0000","last_became_active":"2024-05-09T05:01:05.352787+0000","last_became_peered":"2024-05-09T05:01:05.352787+0000","last_unstale":"2024-05-09T05:01:06.347489+0000","last_undegraded":"2024-05-09T05:01:06.347489+0000","last_fullsized":"2024-05-09T05:01:06.347489+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:12:42.318279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553952+0000","last_change":"2024-05-09T05:01:05.348707+0000","last_active":"2024-05-09T05:01:06.553952+0000","last_peered":"2024-05-09T05:01:06.553952+0000","last_clean":"2024-05-09T05:01:06.553952+0000","last_became_active":"2024-05-09T05:01:05.348447+0000","last_became_peered":"2024-05-09T05:01:05.348447+0000","last_unstale":"2024-05-09T05:01:06.553952+0000","last_undegraded":"2024-05-09T05:01:06.553952+0000","last_fullsized":"2024-05-09T05:01:06.553952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:06:51.148491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.390285+0000","last_change":"2024-05-09T05:01:05.349074+0000","last_active":"2024-05-09T05:01:06.390285+0000","last_peered":"2024-05-09T05:01:06.390285+0000","last_clean":"2024-05-09T05:01:06.390285+0000","last_became_active":"2024-05-09T05:01:05.348803+0000","last_became_peered":"2024-05-09T05:01:05.348803+0000","last_unstale":"2024-05-09T05:01:06.390285+0000","last_undegraded":"2024-05-09T05:01:06.390285+0000","last_fullsized":"2024-05-09T05:01:06.390285+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:39:57.651574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553728+0000","last_change":"2024-05-09T05:01:05.349504+0000","last_active":"2024-05-09T05:01:06.553728+0000","last_peered":"2024-05-09T05:01:06.553728+0000","last_clean":"2024-05-09T05:01:06.553728+0000","last_became_active":"2024-05-09T05:01:05.349187+0000","last_became_peered":"2024-05-09T05:01:05.349187+0000","last_unstale":"2024-05-09T05:01:06.553728+0000","last_undegraded":"2024-05-09T05:01:06.553728+0000","last_fullsized":"2024-05-09T05:01:06.553728+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:56:54.390891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.349962+0000","last_change":"2024-05-09T05:01:05.354923+0000","last_active":"2024-05-09T05:01:06.349962+0000","last_peered":"2024-05-09T05:01:06.349962+0000","last_clean":"2024-05-09T05:01:06.349962+0000","last_became_active":"2024-05-09T05:01:05.354530+0000","last_became_peered":"2024-05-09T05:01:05.354530+0000","last_unstale":"2024-05-09T05:01:06.349962+0000","last_undegraded":"2024-05-09T05:01:06.349962+0000","last_fullsized":"2024-05-09T05:01:06.349962+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:27.412828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553338+0000","last_change":"2024-05-09T05:01:05.349745+0000","last_active":"2024-05-09T05:01:06.553338+0000","last_peered":"2024-05-09T05:01:06.553338+0000","last_clean":"2024-05-09T05:01:06.553338+0000","last_became_active":"2024-05-09T05:01:05.349309+0000","last_became_peered":"2024-05-09T05:01:05.349309+0000","last_unstale":"2024-05-09T05:01:06.553338+0000","last_undegraded":"2024-05-09T05:01:06.553338+0000","last_fullsized":"2024-05-09T05:01:06.553338+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:52:45.792920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553383+0000","last_change":"2024-05-09T05:01:05.352247+0000","last_active":"2024-05-09T05:01:06.553383+0000","last_peered":"2024-05-09T05:01:06.553383+0000","last_clean":"2024-05-09T05:01:06.553383+0000","last_became_active":"2024-05-09T05:01:05.351968+0000","last_became_peered":"2024-05-09T05:01:05.351968+0000","last_unstale":"2024-05-09T05:01:06.553383+0000","last_undegraded":"2024-05-09T05:01:06.553383+0000","last_fullsized":"2024-05-09T05:01:06.553383+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:20.893689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346106+0000","last_change":"2024-05-09T05:01:05.353823+0000","last_active":"2024-05-09T05:01:06.346106+0000","last_peered":"2024-05-09T05:01:06.346106+0000","last_clean":"2024-05-09T05:01:06.346106+0000","last_became_active":"2024-05-09T05:01:05.353577+0000","last_became_peered":"2024-05-09T05:01:05.353577+0000","last_unstale":"2024-05-09T05:01:06.346106+0000","last_undegraded":"2024-05-09T05:01:06.346106+0000","last_fullsized":"2024-05-09T05:01:06.346106+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:04.027664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.389928+0000","last_change":"2024-05-09T05:01:05.352590+0000","last_active":"2024-05-09T05:01:06.389928+0000","last_peered":"2024-05-09T05:01:06.389928+0000","last_clean":"2024-05-09T05:01:06.389928+0000","last_became_active":"2024-05-09T05:01:05.352330+0000","last_became_peered":"2024-05-09T05:01:05.352330+0000","last_unstale":"2024-05-09T05:01:06.389928+0000","last_undegraded":"2024-05-09T05:01:06.389928+0000","last_fullsized":"2024-05-09T05:01:06.389928+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:04:43.523719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346433+0000","last_change":"2024-05-09T05:01:05.350101+0000","last_active":"2024-05-09T05:01:06.346433+0000","last_peered":"2024-05-09T05:01:06.346433+0000","last_clean":"2024-05-09T05:01:06.346433+0000","last_became_active":"2024-05-09T05:01:05.349831+0000","last_became_peered":"2024-05-09T05:01:05.349831+0000","last_unstale":"2024-05-09T05:01:06.346433+0000","last_undegraded":"2024-05-09T05:01:06.346433+0000","last_fullsized":"2024-05-09T05:01:06.346433+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:22:03.591207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"35'4","reported_seq":18,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.554023+0000","last_change":"2024-05-09T05:01:04.339653+0000","last_active":"2024-05-09T05:01:06.554023+0000","last_peered":"2024-05-09T05:01:06.554023+0000","last_clean":"2024-05-09T05:01:06.554023+0000","last_became_active":"2024-05-09T05:01:04.339376+0000","last_became_peered":"2024-05-09T05:01:04.339376+0000","last_unstale":"2024-05-09T05:01:06.554023+0000","last_undegraded":"2024-05-09T05:01:06.554023+0000","last_fullsized":"2024-05-09T05:01:06.554023+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:08.325696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.390162+0000","last_change":"2024-05-09T05:01:05.349287+0000","last_active":"2024-05-09T05:01:06.390162+0000","last_peered":"2024-05-09T05:01:06.390162+0000","last_clean":"2024-05-09T05:01:06.390162+0000","last_became_active":"2024-05-09T05:01:05.349017+0000","last_became_peered":"2024-05-09T05:01:05.349017+0000","last_unstale":"2024-05-09T05:01:06.390162+0000","last_undegraded":"2024-05-09T05:01:06.390162+0000","last_fullsized":"2024-05-09T05:01:06.390162+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:00:27.078945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"35'1","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553031+0000","last_change":"2024-05-09T05:01:04.336251+0000","last_active":"2024-05-09T05:01:06.553031+0000","last_peered":"2024-05-09T05:01:06.553031+0000","last_clean":"2024-05-09T05:01:06.553031+0000","last_became_active":"2024-05-09T05:01:04.335947+0000","last_became_peered":"2024-05-09T05:01:04.335947+0000","last_unstale":"2024-05-09T05:01:06.553031+0000","last_undegraded":"2024-05-09T05:01:06.553031+0000","last_fullsized":"2024-05-09T05:01:06.553031+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:27.200081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.554029+0000","last_change":"2024-05-09T05:01:05.357262+0000","last_active":"2024-05-09T05:01:06.554029+0000","last_peered":"2024-05-09T05:01:06.554029+0000","last_clean":"2024-05-09T05:01:06.554029+0000","last_became_active":"2024-05-09T05:01:05.356588+0000","last_became_peered":"2024-05-09T05:01:05.356588+0000","last_unstale":"2024-05-09T05:01:06.554029+0000","last_undegraded":"2024-05-09T05:01:06.554029+0000","last_fullsized":"2024-05-09T05:01:06.554029+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:04:43.523719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"35'2","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.390188+0000","last_change":"2024-05-09T05:01:04.339576+0000","last_active":"2024-05-09T05:01:06.390188+0000","last_peered":"2024-05-09T05:01:06.390188+0000","last_clean":"2024-05-09T05:01:06.390188+0000","last_became_active":"2024-05-09T05:01:04.339314+0000","last_became_peered":"2024-05-09T05:01:04.339314+0000","last_unstale":"2024-05-09T05:01:06.390188+0000","last_undegraded":"2024-05-09T05:01:06.390188+0000","last_fullsized":"2024-05-09T05:01:06.390188+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:34:23.217355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344791+0000","last_change":"2024-05-09T05:01:05.351578+0000","last_active":"2024-05-09T05:01:06.344791+0000","last_peered":"2024-05-09T05:01:06.344791+0000","last_clean":"2024-05-09T05:01:06.344791+0000","last_became_active":"2024-05-09T05:01:05.351300+0000","last_became_peered":"2024-05-09T05:01:05.351300+0000","last_unstale":"2024-05-09T05:01:06.344791+0000","last_undegraded":"2024-05-09T05:01:06.344791+0000","last_fullsized":"2024-05-09T05:01:06.344791+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:42:57.749068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"35'2","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553777+0000","last_change":"2024-05-09T05:01:04.340150+0000","last_active":"2024-05-09T05:01:06.553777+0000","last_peered":"2024-05-09T05:01:06.553777+0000","last_clean":"2024-05-09T05:01:06.553777+0000","last_became_active":"2024-05-09T05:01:04.339879+0000","last_became_peered":"2024-05-09T05:01:04.339879+0000","last_unstale":"2024-05-09T05:01:06.553777+0000","last_undegraded":"2024-05-09T05:01:06.553777+0000","last_fullsized":"2024-05-09T05:01:06.553777+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:14:27.134789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.552743+0000","last_change":"2024-05-09T05:01:05.348052+0000","last_active":"2024-05-09T05:01:06.552743+0000","last_peered":"2024-05-09T05:01:06.552743+0000","last_clean":"2024-05-09T05:01:06.552743+0000","last_became_active":"2024-05-09T05:01:05.347720+0000","last_became_peered":"2024-05-09T05:01:05.347720+0000","last_unstale":"2024-05-09T05:01:06.552743+0000","last_undegraded":"2024-05-09T05:01:06.552743+0000","last_fullsized":"2024-05-09T05:01:06.552743+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:43:40.752597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"35'1","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.349853+0000","last_change":"2024-05-09T05:01:04.340877+0000","last_active":"2024-05-09T05:01:06.349853+0000","last_peered":"2024-05-09T05:01:06.349853+0000","last_clean":"2024-05-09T05:01:06.349853+0000","last_became_active":"2024-05-09T05:01:04.340598+0000","last_became_peered":"2024-05-09T05:01:04.340598+0000","last_unstale":"2024-05-09T05:01:06.349853+0000","last_undegraded":"2024-05-09T05:01:06.349853+0000","last_fullsized":"2024-05-09T05:01:06.349853+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:56:53.386203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.349812+0000","last_change":"2024-05-09T05:01:05.351056+0000","last_active":"2024-05-09T05:01:06.349812+0000","last_peered":"2024-05-09T05:01:06.349812+0000","last_clean":"2024-05-09T05:01:06.349812+0000","last_became_active":"2024-05-09T05:01:05.350784+0000","last_became_peered":"2024-05-09T05:01:05.350784+0000","last_unstale":"2024-05-09T05:01:06.349812+0000","last_undegraded":"2024-05-09T05:01:06.349812+0000","last_fullsized":"2024-05-09T05:01:06.349812+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:43:59.088798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"35'1","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.345944+0000","last_change":"2024-05-09T05:01:04.341057+0000","last_active":"2024-05-09T05:01:06.345944+0000","last_peered":"2024-05-09T05:01:06.345944+0000","last_clean":"2024-05-09T05:01:06.345944+0000","last_became_active":"2024-05-09T05:01:04.340771+0000","last_became_peered":"2024-05-09T05:01:04.340771+0000","last_unstale":"2024-05-09T05:01:06.345944+0000","last_undegraded":"2024-05-09T05:01:06.345944+0000","last_fullsized":"2024-05-09T05:01:06.345944+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:02:59.742688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553953+0000","last_change":"2024-05-09T05:01:05.354344+0000","last_active":"2024-05-09T05:01:06.553953+0000","last_peered":"2024-05-09T05:01:06.553953+0000","last_clean":"2024-05-09T05:01:06.553953+0000","last_became_active":"2024-05-09T05:01:05.354039+0000","last_became_peered":"2024-05-09T05:01:05.354039+0000","last_unstale":"2024-05-09T05:01:06.553953+0000","last_undegraded":"2024-05-09T05:01:06.553953+0000","last_fullsized":"2024-05-09T05:01:06.553953+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:06:51.148491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.390002+0000","last_change":"2024-05-09T05:01:04.338831+0000","last_active":"2024-05-09T05:01:06.390002+0000","last_peered":"2024-05-09T05:01:06.390002+0000","last_clean":"2024-05-09T05:01:06.390002+0000","last_became_active":"2024-05-09T05:01:04.338565+0000","last_became_peered":"2024-05-09T05:01:04.338565+0000","last_unstale":"2024-05-09T05:01:06.390002+0000","last_undegraded":"2024-05-09T05:01:06.390002+0000","last_fullsized":"2024-05-09T05:01:06.390002+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:04.648238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344468+0000","last_change":"2024-05-09T05:01:05.347212+0000","last_active":"2024-05-09T05:01:06.344468+0000","last_peered":"2024-05-09T05:01:06.344468+0000","last_clean":"2024-05-09T05:01:06.344468+0000","last_became_active":"2024-05-09T05:01:05.346912+0000","last_became_peered":"2024-05-09T05:01:05.346912+0000","last_unstale":"2024-05-09T05:01:06.344468+0000","last_undegraded":"2024-05-09T05:01:06.344468+0000","last_fullsized":"2024-05-09T05:01:06.344468+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:06:27.722335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553379+0000","last_change":"2024-05-09T05:01:04.339770+0000","last_active":"2024-05-09T05:01:06.553379+0000","last_peered":"2024-05-09T05:01:06.553379+0000","last_clean":"2024-05-09T05:01:06.553379+0000","last_became_active":"2024-05-09T05:01:04.339497+0000","last_became_peered":"2024-05-09T05:01:04.339497+0000","last_unstale":"2024-05-09T05:01:06.553379+0000","last_undegraded":"2024-05-09T05:01:06.553379+0000","last_fullsized":"2024-05-09T05:01:06.553379+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:32:59.160180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.347878+0000","last_change":"2024-05-09T05:01:05.357090+0000","last_active":"2024-05-09T05:01:06.347878+0000","last_peered":"2024-05-09T05:01:06.347878+0000","last_clean":"2024-05-09T05:01:06.347878+0000","last_became_active":"2024-05-09T05:01:05.356584+0000","last_became_peered":"2024-05-09T05:01:05.356584+0000","last_unstale":"2024-05-09T05:01:06.347878+0000","last_undegraded":"2024-05-09T05:01:06.347878+0000","last_fullsized":"2024-05-09T05:01:06.347878+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:38:35.684279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344790+0000","last_change":"2024-05-09T05:01:04.341197+0000","last_active":"2024-05-09T05:01:06.344790+0000","last_peered":"2024-05-09T05:01:06.344790+0000","last_clean":"2024-05-09T05:01:06.344790+0000","last_became_active":"2024-05-09T05:01:04.340797+0000","last_became_peered":"2024-05-09T05:01:04.340797+0000","last_unstale":"2024-05-09T05:01:06.344790+0000","last_undegraded":"2024-05-09T05:01:06.344790+0000","last_fullsized":"2024-05-09T05:01:06.344790+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:56:53.386203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346282+0000","last_change":"2024-05-09T05:01:05.347902+0000","last_active":"2024-05-09T05:01:06.346282+0000","last_peered":"2024-05-09T05:01:06.346282+0000","last_clean":"2024-05-09T05:01:06.346282+0000","last_became_active":"2024-05-09T05:01:05.347629+0000","last_became_peered":"2024-05-09T05:01:05.347629+0000","last_unstale":"2024-05-09T05:01:06.346282+0000","last_undegraded":"2024-05-09T05:01:06.346282+0000","last_fullsized":"2024-05-09T05:01:06.346282+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:58.791766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.350132+0000","last_change":"2024-05-09T05:01:04.341543+0000","last_active":"2024-05-09T05:01:06.350132+0000","last_peered":"2024-05-09T05:01:06.350132+0000","last_clean":"2024-05-09T05:01:06.350132+0000","last_became_active":"2024-05-09T05:01:04.341237+0000","last_became_peered":"2024-05-09T05:01:04.341237+0000","last_unstale":"2024-05-09T05:01:06.350132+0000","last_undegraded":"2024-05-09T05:01:06.350132+0000","last_fullsized":"2024-05-09T05:01:06.350132+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:37.948413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.552997+0000","last_change":"2024-05-09T05:01:05.347840+0000","last_active":"2024-05-09T05:01:06.552997+0000","last_peered":"2024-05-09T05:01:06.552997+0000","last_clean":"2024-05-09T05:01:06.552997+0000","last_became_active":"2024-05-09T05:01:05.347382+0000","last_became_peered":"2024-05-09T05:01:05.347382+0000","last_unstale":"2024-05-09T05:01:06.552997+0000","last_undegraded":"2024-05-09T05:01:06.552997+0000","last_fullsized":"2024-05-09T05:01:06.552997+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:52.250650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"35'3","reported_seq":17,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.350146+0000","last_change":"2024-05-09T05:01:04.337652+0000","last_active":"2024-05-09T05:01:06.350146+0000","last_peered":"2024-05-09T05:01:06.350146+0000","last_clean":"2024-05-09T05:01:06.350146+0000","last_became_active":"2024-05-09T05:01:04.337383+0000","last_became_peered":"2024-05-09T05:01:04.337383+0000","last_unstale":"2024-05-09T05:01:06.350146+0000","last_undegraded":"2024-05-09T05:01:06.350146+0000","last_fullsized":"2024-05-09T05:01:06.350146+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:10.992348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.554100+0000","last_change":"2024-05-09T05:01:05.355083+0000","last_active":"2024-05-09T05:01:06.554100+0000","last_peered":"2024-05-09T05:01:06.554100+0000","last_clean":"2024-05-09T05:01:06.554100+0000","last_became_active":"2024-05-09T05:01:05.354816+0000","last_became_peered":"2024-05-09T05:01:05.354816+0000","last_unstale":"2024-05-09T05:01:06.554100+0000","last_undegraded":"2024-05-09T05:01:06.554100+0000","last_fullsized":"2024-05-09T05:01:06.554100+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:43:40.752597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"35'6","reported_seq":20,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344508+0000","last_change":"2024-05-09T05:01:04.335947+0000","last_active":"2024-05-09T05:01:06.344508+0000","last_peered":"2024-05-09T05:01:06.344508+0000","last_clean":"2024-05-09T05:01:06.344508+0000","last_became_active":"2024-05-09T05:01:04.335678+0000","last_became_peered":"2024-05-09T05:01:04.335678+0000","last_unstale":"2024-05-09T05:01:06.344508+0000","last_undegraded":"2024-05-09T05:01:06.344508+0000","last_fullsized":"2024-05-09T05:01:06.344508+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:37.948413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.552797+0000","last_change":"2024-05-09T05:01:05.349002+0000","last_active":"2024-05-09T05:01:06.552797+0000","last_peered":"2024-05-09T05:01:06.552797+0000","last_clean":"2024-05-09T05:01:06.552797+0000","last_became_active":"2024-05-09T05:01:05.348746+0000","last_became_peered":"2024-05-09T05:01:05.348746+0000","last_unstale":"2024-05-09T05:01:06.552797+0000","last_undegraded":"2024-05-09T05:01:06.552797+0000","last_fullsized":"2024-05-09T05:01:06.552797+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:02:46.125356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"35'2","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344635+0000","last_change":"2024-05-09T05:01:04.344726+0000","last_active":"2024-05-09T05:01:06.344635+0000","last_peered":"2024-05-09T05:01:06.344635+0000","last_clean":"2024-05-09T05:01:06.344635+0000","last_became_active":"2024-05-09T05:01:04.344384+0000","last_became_peered":"2024-05-09T05:01:04.344384+0000","last_unstale":"2024-05-09T05:01:06.344635+0000","last_undegraded":"2024-05-09T05:01:06.344635+0000","last_fullsized":"2024-05-09T05:01:06.344635+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:06:50.143802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344460+0000","last_change":"2024-05-09T05:01:05.340042+0000","last_active":"2024-05-09T05:01:06.344460+0000","last_peered":"2024-05-09T05:01:06.344460+0000","last_clean":"2024-05-09T05:01:06.344460+0000","last_became_active":"2024-05-09T05:01:05.339742+0000","last_became_peered":"2024-05-09T05:01:05.339742+0000","last_unstale":"2024-05-09T05:01:06.344460+0000","last_undegraded":"2024-05-09T05:01:06.344460+0000","last_fullsized":"2024-05-09T05:01:06.344460+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:18:38.102212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.389909+0000","last_change":"2024-05-09T05:01:04.345746+0000","last_active":"2024-05-09T05:01:06.389909+0000","last_peered":"2024-05-09T05:01:06.389909+0000","last_clean":"2024-05-09T05:01:06.389909+0000","last_became_active":"2024-05-09T05:01:04.345514+0000","last_became_peered":"2024-05-09T05:01:04.345514+0000","last_unstale":"2024-05-09T05:01:06.389909+0000","last_undegraded":"2024-05-09T05:01:06.389909+0000","last_fullsized":"2024-05-09T05:01:06.389909+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:19.502728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346015+0000","last_change":"2024-05-09T05:01:05.353988+0000","last_active":"2024-05-09T05:01:06.346015+0000","last_peered":"2024-05-09T05:01:06.346015+0000","last_clean":"2024-05-09T05:01:06.346015+0000","last_became_active":"2024-05-09T05:01:05.353679+0000","last_became_peered":"2024-05-09T05:01:05.353679+0000","last_unstale":"2024-05-09T05:01:06.346015+0000","last_undegraded":"2024-05-09T05:01:06.346015+0000","last_fullsized":"2024-05-09T05:01:06.346015+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:45:56.221913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553160+0000","last_change":"2024-05-09T05:01:04.337903+0000","last_active":"2024-05-09T05:01:06.553160+0000","last_peered":"2024-05-09T05:01:06.553160+0000","last_clean":"2024-05-09T05:01:06.553160+0000","last_became_active":"2024-05-09T05:01:04.337616+0000","last_became_peered":"2024-05-09T05:01:04.337616+0000","last_unstale":"2024-05-09T05:01:06.553160+0000","last_undegraded":"2024-05-09T05:01:06.553160+0000","last_fullsized":"2024-05-09T05:01:06.553160+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:14:27.134789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344520+0000","last_change":"2024-05-09T05:01:05.340539+0000","last_active":"2024-05-09T05:01:06.344520+0000","last_peered":"2024-05-09T05:01:06.344520+0000","last_clean":"2024-05-09T05:01:06.344520+0000","last_became_active":"2024-05-09T05:01:05.340233+0000","last_became_peered":"2024-05-09T05:01:05.340233+0000","last_unstale":"2024-05-09T05:01:06.344520+0000","last_undegraded":"2024-05-09T05:01:06.344520+0000","last_fullsized":"2024-05-09T05:01:06.344520+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:58.791766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"35'8","reported_seq":22,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.345791+0000","last_change":"2024-05-09T05:01:04.338086+0000","last_active":"2024-05-09T05:01:06.345791+0000","last_peered":"2024-05-09T05:01:06.345791+0000","last_clean":"2024-05-09T05:01:06.345791+0000","last_became_active":"2024-05-09T05:01:04.337822+0000","last_became_peered":"2024-05-09T05:01:04.337822+0000","last_unstale":"2024-05-09T05:01:06.345791+0000","last_undegraded":"2024-05-09T05:01:06.345791+0000","last_fullsized":"2024-05-09T05:01:06.345791+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:34:23.217355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344529+0000","last_change":"2024-05-09T05:01:05.350366+0000","last_active":"2024-05-09T05:01:06.344529+0000","last_peered":"2024-05-09T05:01:06.344529+0000","last_clean":"2024-05-09T05:01:06.344529+0000","last_became_active":"2024-05-09T05:01:05.350068+0000","last_became_peered":"2024-05-09T05:01:05.350068+0000","last_unstale":"2024-05-09T05:01:06.344529+0000","last_undegraded":"2024-05-09T05:01:06.344529+0000","last_fullsized":"2024-05-09T05:01:06.344529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:22:03.591207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"35'1","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.350089+0000","last_change":"2024-05-09T05:01:04.339855+0000","last_active":"2024-05-09T05:01:06.350089+0000","last_peered":"2024-05-09T05:01:06.350089+0000","last_clean":"2024-05-09T05:01:06.350089+0000","last_became_active":"2024-05-09T05:01:04.339601+0000","last_became_peered":"2024-05-09T05:01:04.339601+0000","last_unstale":"2024-05-09T05:01:06.350089+0000","last_undegraded":"2024-05-09T05:01:06.350089+0000","last_fullsized":"2024-05-09T05:01:06.350089+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:52:44.788232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.350081+0000","last_change":"2024-05-09T05:01:05.341841+0000","last_active":"2024-05-09T05:01:06.350081+0000","last_peered":"2024-05-09T05:01:06.350081+0000","last_clean":"2024-05-09T05:01:06.350081+0000","last_became_active":"2024-05-09T05:01:05.341643+0000","last_became_peered":"2024-05-09T05:01:05.341643+0000","last_unstale":"2024-05-09T05:01:06.350081+0000","last_undegraded":"2024-05-09T05:01:06.350081+0000","last_fullsized":"2024-05-09T05:01:06.350081+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:29:27.988279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"35'2","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.554067+0000","last_change":"2024-05-09T05:01:04.338411+0000","last_active":"2024-05-09T05:01:06.554067+0000","last_peered":"2024-05-09T05:01:06.554067+0000","last_clean":"2024-05-09T05:01:06.554067+0000","last_became_active":"2024-05-09T05:01:04.338123+0000","last_became_peered":"2024-05-09T05:01:04.338123+0000","last_unstale":"2024-05-09T05:01:06.554067+0000","last_undegraded":"2024-05-09T05:01:06.554067+0000","last_fullsized":"2024-05-09T05:01:06.554067+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:08.325696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553953+0000","last_change":"2024-05-09T05:01:05.356091+0000","last_active":"2024-05-09T05:01:06.553953+0000","last_peered":"2024-05-09T05:01:06.553953+0000","last_clean":"2024-05-09T05:01:06.553953+0000","last_became_active":"2024-05-09T05:01:05.355752+0000","last_became_peered":"2024-05-09T05:01:05.355752+0000","last_unstale":"2024-05-09T05:01:06.553953+0000","last_undegraded":"2024-05-09T05:01:06.553953+0000","last_fullsized":"2024-05-09T05:01:06.553953+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:56:54.390891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"35'1","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.347944+0000","last_change":"2024-05-09T05:01:04.336039+0000","last_active":"2024-05-09T05:01:06.347944+0000","last_peered":"2024-05-09T05:01:06.347944+0000","last_clean":"2024-05-09T05:01:06.347944+0000","last_became_active":"2024-05-09T05:01:04.335735+0000","last_became_peered":"2024-05-09T05:01:04.335735+0000","last_unstale":"2024-05-09T05:01:06.347944+0000","last_undegraded":"2024-05-09T05:01:06.347944+0000","last_fullsized":"2024-05-09T05:01:06.347944+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:58.290034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.348127+0000","last_change":"2024-05-09T05:01:05.358170+0000","last_active":"2024-05-09T05:01:06.348127+0000","last_peered":"2024-05-09T05:01:06.348127+0000","last_clean":"2024-05-09T05:01:06.348127+0000","last_became_active":"2024-05-09T05:01:05.357840+0000","last_became_peered":"2024-05-09T05:01:05.357840+0000","last_unstale":"2024-05-09T05:01:06.348127+0000","last_undegraded":"2024-05-09T05:01:06.348127+0000","last_fullsized":"2024-05-09T05:01:06.348127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:33:00.164869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.390035+0000","last_change":"2024-05-09T05:01:04.340174+0000","last_active":"2024-05-09T05:01:06.390035+0000","last_peered":"2024-05-09T05:01:06.390035+0000","last_clean":"2024-05-09T05:01:06.390035+0000","last_became_active":"2024-05-09T05:01:04.339896+0000","last_became_peered":"2024-05-09T05:01:04.339896+0000","last_unstale":"2024-05-09T05:01:06.390035+0000","last_undegraded":"2024-05-09T05:01:06.390035+0000","last_fullsized":"2024-05-09T05:01:06.390035+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:27.200081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346219+0000","last_change":"2024-05-09T05:01:05.354792+0000","last_active":"2024-05-09T05:01:06.346219+0000","last_peered":"2024-05-09T05:01:06.346219+0000","last_clean":"2024-05-09T05:01:06.346219+0000","last_became_active":"2024-05-09T05:01:05.354395+0000","last_became_peered":"2024-05-09T05:01:05.354395+0000","last_unstale":"2024-05-09T05:01:06.346219+0000","last_undegraded":"2024-05-09T05:01:06.346219+0000","last_fullsized":"2024-05-09T05:01:06.346219+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:06:27.722335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"35'2","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.347533+0000","last_change":"2024-05-09T05:01:04.337165+0000","last_active":"2024-05-09T05:01:06.347533+0000","last_peered":"2024-05-09T05:01:06.347533+0000","last_clean":"2024-05-09T05:01:06.347533+0000","last_became_active":"2024-05-09T05:01:04.336861+0000","last_became_peered":"2024-05-09T05:01:04.336861+0000","last_unstale":"2024-05-09T05:01:06.347533+0000","last_undegraded":"2024-05-09T05:01:06.347533+0000","last_fullsized":"2024-05-09T05:01:06.347533+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:49.639516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.552803+0000","last_change":"2024-05-09T05:01:05.345491+0000","last_active":"2024-05-09T05:01:06.552803+0000","last_peered":"2024-05-09T05:01:06.552803+0000","last_clean":"2024-05-09T05:01:06.552803+0000","last_became_active":"2024-05-09T05:01:05.345155+0000","last_became_peered":"2024-05-09T05:01:05.345155+0000","last_unstale":"2024-05-09T05:01:06.552803+0000","last_undegraded":"2024-05-09T05:01:06.552803+0000","last_fullsized":"2024-05-09T05:01:06.552803+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:36:28.104239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.347515+0000","last_change":"2024-05-09T05:01:04.338378+0000","last_active":"2024-05-09T05:01:06.347515+0000","last_peered":"2024-05-09T05:01:06.347515+0000","last_clean":"2024-05-09T05:01:06.347515+0000","last_became_active":"2024-05-09T05:01:04.338002+0000","last_became_peered":"2024-05-09T05:01:04.338002+0000","last_unstale":"2024-05-09T05:01:06.347515+0000","last_undegraded":"2024-05-09T05:01:06.347515+0000","last_fullsized":"2024-05-09T05:01:06.347515+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:23:07.883786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.349723+0000","last_change":"2024-05-09T05:01:05.347833+0000","last_active":"2024-05-09T05:01:06.349723+0000","last_peered":"2024-05-09T05:01:06.349723+0000","last_clean":"2024-05-09T05:01:06.349723+0000","last_became_active":"2024-05-09T05:01:05.346828+0000","last_became_peered":"2024-05-09T05:01:05.346828+0000","last_unstale":"2024-05-09T05:01:06.349723+0000","last_undegraded":"2024-05-09T05:01:06.349723+0000","last_fullsized":"2024-05-09T05:01:06.349723+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:58.791766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346070+0000","last_change":"2024-05-09T05:01:04.339021+0000","last_active":"2024-05-09T05:01:06.346070+0000","last_peered":"2024-05-09T05:01:06.346070+0000","last_clean":"2024-05-09T05:01:06.346070+0000","last_became_active":"2024-05-09T05:01:04.338750+0000","last_became_peered":"2024-05-09T05:01:04.338750+0000","last_unstale":"2024-05-09T05:01:06.346070+0000","last_undegraded":"2024-05-09T05:01:06.346070+0000","last_fullsized":"2024-05-09T05:01:06.346070+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:52:44.788232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344651+0000","last_change":"2024-05-09T05:01:05.347210+0000","last_active":"2024-05-09T05:01:06.344651+0000","last_peered":"2024-05-09T05:01:06.344651+0000","last_clean":"2024-05-09T05:01:06.344651+0000","last_became_active":"2024-05-09T05:01:05.346891+0000","last_became_peered":"2024-05-09T05:01:05.346891+0000","last_unstale":"2024-05-09T05:01:06.344651+0000","last_undegraded":"2024-05-09T05:01:06.344651+0000","last_fullsized":"2024-05-09T05:01:06.344651+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:43:59.088798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.347543+0000","last_change":"2024-05-09T05:01:04.336987+0000","last_active":"2024-05-09T05:01:06.347543+0000","last_peered":"2024-05-09T05:01:06.347543+0000","last_clean":"2024-05-09T05:01:06.347543+0000","last_became_active":"2024-05-09T05:01:04.336585+0000","last_became_peered":"2024-05-09T05:01:04.336585+0000","last_unstale":"2024-05-09T05:01:06.347543+0000","last_undegraded":"2024-05-09T05:01:06.347543+0000","last_fullsized":"2024-05-09T05:01:06.347543+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:32:47.893689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.390017+0000","last_change":"2024-05-09T05:01:05.348685+0000","last_active":"2024-05-09T05:01:06.390017+0000","last_peered":"2024-05-09T05:01:06.390017+0000","last_clean":"2024-05-09T05:01:06.390017+0000","last_became_active":"2024-05-09T05:01:05.348374+0000","last_became_peered":"2024-05-09T05:01:05.348374+0000","last_unstale":"2024-05-09T05:01:06.390017+0000","last_undegraded":"2024-05-09T05:01:06.390017+0000","last_fullsized":"2024-05-09T05:01:06.390017+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:02:46.125356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"35'2","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.350165+0000","last_change":"2024-05-09T05:01:04.342282+0000","last_active":"2024-05-09T05:01:06.350165+0000","last_peered":"2024-05-09T05:01:06.350165+0000","last_clean":"2024-05-09T05:01:06.350165+0000","last_became_active":"2024-05-09T05:01:04.342025+0000","last_became_peered":"2024-05-09T05:01:04.342025+0000","last_unstale":"2024-05-09T05:01:06.350165+0000","last_undegraded":"2024-05-09T05:01:06.350165+0000","last_fullsized":"2024-05-09T05:01:06.350165+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:19.889000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346609+0000","last_change":"2024-05-09T05:01:05.348627+0000","last_active":"2024-05-09T05:01:06.346609+0000","last_peered":"2024-05-09T05:01:06.346609+0000","last_clean":"2024-05-09T05:01:06.346609+0000","last_became_active":"2024-05-09T05:01:05.348330+0000","last_became_peered":"2024-05-09T05:01:05.348330+0000","last_unstale":"2024-05-09T05:01:06.346609+0000","last_undegraded":"2024-05-09T05:01:06.346609+0000","last_fullsized":"2024-05-09T05:01:06.346609+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:51:54.638779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"36'1","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.349142+0000","last_change":"2024-05-09T05:01:04.337098+0000","last_active":"2024-05-09T05:01:06.349142+0000","last_peered":"2024-05-09T05:01:06.349142+0000","last_clean":"2024-05-09T05:01:06.349142+0000","last_became_active":"2024-05-09T05:01:04.336769+0000","last_became_peered":"2024-05-09T05:01:04.336769+0000","last_unstale":"2024-05-09T05:01:06.349142+0000","last_undegraded":"2024-05-09T05:01:06.349142+0000","last_fullsized":"2024-05-09T05:01:06.349142+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:47:49.674854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344675+0000","last_change":"2024-05-09T05:01:05.357533+0000","last_active":"2024-05-09T05:01:06.344675+0000","last_peered":"2024-05-09T05:01:06.344675+0000","last_clean":"2024-05-09T05:01:06.344675+0000","last_became_active":"2024-05-09T05:01:05.357200+0000","last_became_peered":"2024-05-09T05:01:05.357200+0000","last_unstale":"2024-05-09T05:01:06.344675+0000","last_undegraded":"2024-05-09T05:01:06.344675+0000","last_fullsized":"2024-05-09T05:01:06.344675+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:27.412828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553043+0000","last_change":"2024-05-09T05:01:04.334350+0000","last_active":"2024-05-09T05:01:06.553043+0000","last_peered":"2024-05-09T05:01:06.553043+0000","last_clean":"2024-05-09T05:01:06.553043+0000","last_became_active":"2024-05-09T05:01:04.334089+0000","last_became_peered":"2024-05-09T05:01:04.334089+0000","last_unstale":"2024-05-09T05:01:06.553043+0000","last_undegraded":"2024-05-09T05:01:06.553043+0000","last_fullsized":"2024-05-09T05:01:06.553043+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:04.648238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.552911+0000","last_change":"2024-05-09T05:01:05.351990+0000","last_active":"2024-05-09T05:01:06.552911+0000","last_peered":"2024-05-09T05:01:06.552911+0000","last_clean":"2024-05-09T05:01:06.552911+0000","last_became_active":"2024-05-09T05:01:05.351720+0000","last_became_peered":"2024-05-09T05:01:05.351720+0000","last_unstale":"2024-05-09T05:01:06.552911+0000","last_undegraded":"2024-05-09T05:01:06.552911+0000","last_fullsized":"2024-05-09T05:01:06.552911+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:00:27.078945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.552965+0000","last_change":"2024-05-09T05:01:04.342024+0000","last_active":"2024-05-09T05:01:06.552965+0000","last_peered":"2024-05-09T05:01:06.552965+0000","last_clean":"2024-05-09T05:01:06.552965+0000","last_became_active":"2024-05-09T05:01:04.341764+0000","last_became_peered":"2024-05-09T05:01:04.341764+0000","last_unstale":"2024-05-09T05:01:06.552965+0000","last_undegraded":"2024-05-09T05:01:06.552965+0000","last_fullsized":"2024-05-09T05:01:06.552965+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:19.502728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.389978+0000","last_change":"2024-05-09T05:01:05.346758+0000","last_active":"2024-05-09T05:01:06.389978+0000","last_peered":"2024-05-09T05:01:06.389978+0000","last_clean":"2024-05-09T05:01:06.389978+0000","last_became_active":"2024-05-09T05:01:05.346442+0000","last_became_peered":"2024-05-09T05:01:05.346442+0000","last_unstale":"2024-05-09T05:01:06.389978+0000","last_undegraded":"2024-05-09T05:01:06.389978+0000","last_fullsized":"2024-05-09T05:01:06.389978+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:43:40.752597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553656+0000","last_change":"2024-05-09T05:01:04.330105+0000","last_active":"2024-05-09T05:01:06.553656+0000","last_peered":"2024-05-09T05:01:06.553656+0000","last_clean":"2024-05-09T05:01:06.553656+0000","last_became_active":"2024-05-09T05:01:04.329841+0000","last_became_peered":"2024-05-09T05:01:04.329841+0000","last_unstale":"2024-05-09T05:01:06.553656+0000","last_undegraded":"2024-05-09T05:01:06.553656+0000","last_fullsized":"2024-05-09T05:01:06.553656+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:32:59.160180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553757+0000","last_change":"2024-05-09T05:01:05.358864+0000","last_active":"2024-05-09T05:01:06.553757+0000","last_peered":"2024-05-09T05:01:06.553757+0000","last_clean":"2024-05-09T05:01:06.553757+0000","last_became_active":"2024-05-09T05:01:05.358593+0000","last_became_peered":"2024-05-09T05:01:05.358593+0000","last_unstale":"2024-05-09T05:01:06.553757+0000","last_undegraded":"2024-05-09T05:01:06.553757+0000","last_fullsized":"2024-05-09T05:01:06.553757+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:11.997036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553710+0000","last_change":"2024-05-09T05:01:04.340828+0000","last_active":"2024-05-09T05:01:06.553710+0000","last_peered":"2024-05-09T05:01:06.553710+0000","last_clean":"2024-05-09T05:01:06.553710+0000","last_became_active":"2024-05-09T05:01:04.340564+0000","last_became_peered":"2024-05-09T05:01:04.340564+0000","last_unstale":"2024-05-09T05:01:06.553710+0000","last_undegraded":"2024-05-09T05:01:06.553710+0000","last_fullsized":"2024-05-09T05:01:06.553710+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:38:34.679590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346195+0000","last_change":"2024-05-09T05:01:05.348824+0000","last_active":"2024-05-09T05:01:06.346195+0000","last_peered":"2024-05-09T05:01:06.346195+0000","last_clean":"2024-05-09T05:01:06.346195+0000","last_became_active":"2024-05-09T05:01:05.348552+0000","last_became_peered":"2024-05-09T05:01:05.348552+0000","last_unstale":"2024-05-09T05:01:06.346195+0000","last_undegraded":"2024-05-09T05:01:06.346195+0000","last_fullsized":"2024-05-09T05:01:06.346195+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:18:38.102212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344179+0000","last_change":"2024-05-09T05:01:04.341097+0000","last_active":"2024-05-09T05:01:06.344179+0000","last_peered":"2024-05-09T05:01:06.344179+0000","last_clean":"2024-05-09T05:01:06.344179+0000","last_became_active":"2024-05-09T05:01:04.340782+0000","last_became_peered":"2024-05-09T05:01:04.340782+0000","last_unstale":"2024-05-09T05:01:06.344179+0000","last_undegraded":"2024-05-09T05:01:06.344179+0000","last_fullsized":"2024-05-09T05:01:06.344179+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:10.992348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.350065+0000","last_change":"2024-05-09T05:01:05.356576+0000","last_active":"2024-05-09T05:01:06.350065+0000","last_peered":"2024-05-09T05:01:06.350065+0000","last_clean":"2024-05-09T05:01:06.350065+0000","last_became_active":"2024-05-09T05:01:05.356274+0000","last_became_peered":"2024-05-09T05:01:05.356274+0000","last_unstale":"2024-05-09T05:01:06.350065+0000","last_undegraded":"2024-05-09T05:01:06.350065+0000","last_fullsized":"2024-05-09T05:01:06.350065+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:28.747938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346341+0000","last_change":"2024-05-09T05:01:04.336368+0000","last_active":"2024-05-09T05:01:06.346341+0000","last_peered":"2024-05-09T05:01:06.346341+0000","last_clean":"2024-05-09T05:01:06.346341+0000","last_became_active":"2024-05-09T05:01:04.336092+0000","last_became_peered":"2024-05-09T05:01:04.336092+0000","last_unstale":"2024-05-09T05:01:06.346341+0000","last_undegraded":"2024-05-09T05:01:06.346341+0000","last_fullsized":"2024-05-09T05:01:06.346341+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:19.889000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":139,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346303+0000","last_change":"2024-05-09T05:00:08.667158+0000","last_active":"2024-05-09T05:01:06.346303+0000","last_peered":"2024-05-09T05:01:06.346303+0000","last_clean":"2024-05-09T05:01:06.346303+0000","last_became_active":"2024-05-09T05:00:08.666316+0000","last_became_peered":"2024-05-09T05:00:08.666316+0000","last_unstale":"2024-05-09T05:01:06.346303+0000","last_undegraded":"2024-05-09T05:01:06.346303+0000","last_fullsized":"2024-05-09T05:01:06.346303+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_clean_scrub_stamp":"2024-05-09T05:00:07.370119+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:20.234643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574862,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343828,"statfs":{"total":96636764160,"available":96608079872,"internally_reserved":0,"allocated":1011712,"data_stored":760669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574863,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":169201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574863,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":168731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":169223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574862,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":417792,"data_stored":170265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574862,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":417792,"data_stored":175381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":417792,"data_stored":175401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1011712,"data_stored":765713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:01:09.388 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:01:09.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.527+0000 7feb3fc25640 1 Processor -- start 2024-05-09T05:01:09.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.527+0000 7feb3fc25640 1 -- start start 2024-05-09T05:01:09.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.528+0000 7feb3fc25640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb381606f0 0x7feb380d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:09.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.528+0000 7feb3fc25640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7feb3806f9f0 0x7feb380d3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:09.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.528+0000 7feb3fc25640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb3806ff60 con 0x7feb3815f300 2024-05-09T05:01:09.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.528+0000 7feb3fc25640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7feb38070240 con 0x7feb3806f9f0 2024-05-09T05:01:09.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.528+0000 7feb3fc25640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7feb38070520 con 0x7feb381606f0 2024-05-09T05:01:09.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.528+0000 7feb3e1c2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7feb3806f9f0 0x7feb380d3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:09.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.528+0000 7feb3e1c2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7feb3806f9f0 0x7feb380d3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59794/0 (socket says 172.21.15.60:59794) 2024-05-09T05:01:09.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.528+0000 7feb3e1c2640 1 -- 172.21.15.60:0/2203551362 learned_addr learned my addr 172.21.15.60:0/2203551362 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:09.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.528+0000 7feb3e1c2640 1 -- 172.21.15.60:0/2203551362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb381606f0 msgr2=0x7feb380d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:09.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.528+0000 7feb3d1c0640 1 --2- 172.21.15.60:0/2203551362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb381606f0 0x7feb380d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:09.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.528+0000 7feb3e1c2640 1 --2- 172.21.15.60:0/2203551362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb381606f0 0x7feb380d33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.528+0000 7feb3e1c2640 1 -- 172.21.15.60:0/2203551362 >> v1:172.21.15.114:6789/0 conn(0x7feb3815f300 legacy=0x7feb381639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:09.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.528+0000 7feb3e1c2640 1 -- 172.21.15.60:0/2203551362 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb3806e740 con 0x7feb3806f9f0 2024-05-09T05:01:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.529+0000 7feb3e1c2640 1 --2- 172.21.15.60:0/2203551362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb3806f9f0 0x7feb380d3930 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7feb30009e50 tx=0x7feb3002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d80984df54e626 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.529+0000 7feb3c9bf640 1 -- 172.21.15.60:0/2203551362 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb30036070 con 0x7feb3806f9f0 2024-05-09T05:01:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.529+0000 7feb3c9bf640 1 -- 172.21.15.60:0/2203551362 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb30031600 con 0x7feb3806f9f0 2024-05-09T05:01:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.529+0000 7feb3c9bf640 1 -- 172.21.15.60:0/2203551362 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb300319d0 con 0x7feb3806f9f0 2024-05-09T05:01:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.530+0000 7feb3fc25640 1 -- 172.21.15.60:0/2203551362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb3806f9f0 msgr2=0x7feb380d3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.530+0000 7feb3fc25640 1 --2- 172.21.15.60:0/2203551362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb3806f9f0 0x7feb380d3930 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7feb30009e50 tx=0x7feb3002a680 comp rx=0 tx=0).stop 2024-05-09T05:01:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.530+0000 7feb3fc25640 1 -- 172.21.15.60:0/2203551362 shutdown_connections 2024-05-09T05:01:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.530+0000 7feb3fc25640 1 --2- 172.21.15.60:0/2203551362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb3806f9f0 0x7feb380d3930 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.530+0000 7feb3fc25640 1 --2- 172.21.15.60:0/2203551362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb381606f0 0x7feb380d33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.530+0000 7feb3fc25640 1 -- 172.21.15.60:0/2203551362 >> 172.21.15.60:0/2203551362 conn(0x7feb3809b850 msgr2=0x7feb3809bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.530+0000 7feb3fc25640 1 -- 172.21.15.60:0/2203551362 shutdown_connections 2024-05-09T05:01:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.530+0000 7feb3fc25640 1 -- 172.21.15.60:0/2203551362 wait complete. 2024-05-09T05:01:09.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.531+0000 7feb3fc25640 1 Processor -- start 2024-05-09T05:01:09.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.531+0000 7feb3fc25640 1 -- start start 2024-05-09T05:01:09.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.532+0000 7feb3fc25640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb3806f9f0 0x7feb38163390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:09.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.532+0000 7feb3fc25640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb3815f300 0x7feb381f9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:09.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.532+0000 7feb3fc25640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb381606f0 0x7feb3815de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:09.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.532+0000 7feb3fc25640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7feb3806f0c0 con 0x7feb3815f300 2024-05-09T05:01:09.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.532+0000 7feb3fc25640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7feb3806ef40 con 0x7feb3806f9f0 2024-05-09T05:01:09.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.532+0000 7feb3fc25640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7feb3806f240 con 0x7feb381606f0 2024-05-09T05:01:09.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.532+0000 7feb3d1c0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb3815f300 0x7feb381f9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:09.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.532+0000 7feb3d1c0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb3815f300 0x7feb381f9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59796/0 (socket says 172.21.15.60:59796) 2024-05-09T05:01:09.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.532+0000 7feb3d1c0640 1 -- 172.21.15.60:0/1268085520 learned_addr learned my addr 172.21.15.60:0/1268085520 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:09.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.532+0000 7feb3d9c1640 1 --2- 172.21.15.60:0/1268085520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb3806f9f0 0x7feb38163390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:09.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.532+0000 7feb3e1c2640 1 --2- 172.21.15.60:0/1268085520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb381606f0 0x7feb3815de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:09.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.532+0000 7feb3d1c0640 1 -- 172.21.15.60:0/1268085520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb381606f0 msgr2=0x7feb3815de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:09.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.532+0000 7feb3d1c0640 1 --2- 172.21.15.60:0/1268085520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb381606f0 0x7feb3815de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.532+0000 7feb3d1c0640 1 -- 172.21.15.60:0/1268085520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb3806f9f0 msgr2=0x7feb38163390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:09.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.532+0000 7feb3d1c0640 1 --2- 172.21.15.60:0/1268085520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb3806f9f0 0x7feb38163390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.532+0000 7feb3d1c0640 1 -- 172.21.15.60:0/1268085520 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb381fa7c0 con 0x7feb3815f300 2024-05-09T05:01:09.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.532+0000 7feb3d1c0640 1 --2- 172.21.15.60:0/1268085520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb3815f300 0x7feb381f9800 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7feb2800c7f0 tx=0x7feb2800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:09.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.532+0000 7feb2e7fc640 1 -- 172.21.15.60:0/1268085520 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb28015070 con 0x7feb3815f300 2024-05-09T05:01:09.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.533+0000 7feb2e7fc640 1 -- 172.21.15.60:0/1268085520 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb28004220 con 0x7feb3815f300 2024-05-09T05:01:09.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.533+0000 7feb3fc25640 1 -- 172.21.15.60:0/1268085520 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb381fa200 con 0x7feb3815f300 2024-05-09T05:01:09.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.533+0000 7feb2e7fc640 1 -- 172.21.15.60:0/1268085520 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb28007620 con 0x7feb3815f300 2024-05-09T05:01:09.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.533+0000 7feb3fc25640 1 -- 172.21.15.60:0/1268085520 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb380ad3e0 con 0x7feb3815f300 2024-05-09T05:01:09.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.533+0000 7feb3fc25640 1 -- 172.21.15.60:0/1268085520 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb381fab20 con 0x7feb3815f300 2024-05-09T05:01:09.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.534+0000 7feb2e7fc640 1 -- 172.21.15.60:0/1268085520 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7feb280077c0 con 0x7feb3815f300 2024-05-09T05:01:09.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.535+0000 7feb2e7fc640 1 --2- 172.21.15.60:0/1268085520 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7feb1c0762d0 0x7feb1c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:09.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.535+0000 7feb2e7fc640 1 -- 172.21.15.60:0/1268085520 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7feb280977c0 con 0x7feb3815f300 2024-05-09T05:01:09.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.535+0000 7feb3d9c1640 1 --2- 172.21.15.60:0/1268085520 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7feb1c0762d0 0x7feb1c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:09.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.536+0000 7feb3d9c1640 1 --2- 172.21.15.60:0/1268085520 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7feb1c0762d0 0x7feb1c078790 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feb34004f60 tx=0x7feb34007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:09.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.539+0000 7feb2e7fc640 1 -- 172.21.15.60:0/1268085520 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7feb280609d0 con 0x7feb3815f300 2024-05-09T05:01:09.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.726+0000 7feb3fc25640 1 -- 172.21.15.60:0/1268085520 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7feb38051040 con 0x7feb1c0762d0 2024-05-09T05:01:09.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.733+0000 7feb2e7fc640 1 -- 172.21.15.60:0/1268085520 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251944 (secure 0 0 0) 0x7feb38051040 con 0x7feb1c0762d0 2024-05-09T05:01:09.737 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:09.738 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-05-09T05:01:09.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.736+0000 7feb3fc25640 1 -- 172.21.15.60:0/1268085520 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7feb1c0762d0 msgr2=0x7feb1c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:09.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.736+0000 7feb3fc25640 1 --2- 172.21.15.60:0/1268085520 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7feb1c0762d0 0x7feb1c078790 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feb34004f60 tx=0x7feb34007e30 comp rx=0 tx=0).stop 2024-05-09T05:01:09.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.736+0000 7feb3fc25640 1 -- 172.21.15.60:0/1268085520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb3815f300 msgr2=0x7feb381f9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:09.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.736+0000 7feb3fc25640 1 --2- 172.21.15.60:0/1268085520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb3815f300 0x7feb381f9800 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7feb2800c7f0 tx=0x7feb2800ccb0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.736+0000 7feb3fc25640 1 -- 172.21.15.60:0/1268085520 shutdown_connections 2024-05-09T05:01:09.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.736+0000 7feb3fc25640 1 --2- 172.21.15.60:0/1268085520 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7feb1c0762d0 0x7feb1c078790 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.736+0000 7feb3fc25640 1 --2- 172.21.15.60:0/1268085520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb381606f0 0x7feb3815de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.736+0000 7feb3fc25640 1 --2- 172.21.15.60:0/1268085520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb3815f300 0x7feb381f9800 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.736+0000 7feb3fc25640 1 --2- 172.21.15.60:0/1268085520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb3806f9f0 0x7feb38163390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.736+0000 7feb3fc25640 1 -- 172.21.15.60:0/1268085520 >> 172.21.15.60:0/1268085520 conn(0x7feb3809b850 msgr2=0x7feb38099ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:09.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.736+0000 7feb3fc25640 1 -- 172.21.15.60:0/1268085520 shutdown_connections 2024-05-09T05:01:09.740 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.736+0000 7feb3fc25640 1 -- 172.21.15.60:0/1268085520 wait complete. 2024-05-09T05:01:09.748 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-05-09T05:01:08.923907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237216,"kb_used_data":4396,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737504,"statfs":{"total":773094113280,"available":772851204096,"internally_reserved":0,"allocated":4501504,"data_stored":2554584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.605477"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344743+0000","last_change":"2024-05-09T05:01:05.342734+0000","last_active":"2024-05-09T05:01:06.344743+0000","last_peered":"2024-05-09T05:01:06.344743+0000","last_clean":"2024-05-09T05:01:06.344743+0000","last_became_active":"2024-05-09T05:01:05.342459+0000","last_became_peered":"2024-05-09T05:01:05.342459+0000","last_unstale":"2024-05-09T05:01:06.344743+0000","last_undegraded":"2024-05-09T05:01:06.344743+0000","last_fullsized":"2024-05-09T05:01:06.344743+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:18:04.101345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344653+0000","last_change":"2024-05-09T05:01:05.349559+0000","last_active":"2024-05-09T05:01:06.344653+0000","last_peered":"2024-05-09T05:01:06.344653+0000","last_clean":"2024-05-09T05:01:06.344653+0000","last_became_active":"2024-05-09T05:01:05.349207+0000","last_became_peered":"2024-05-09T05:01:05.349207+0000","last_unstale":"2024-05-09T05:01:06.344653+0000","last_undegraded":"2024-05-09T05:01:06.344653+0000","last_fullsized":"2024-05-09T05:01:06.344653+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:51:54.638779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.552957+0000","last_change":"2024-05-09T05:01:05.348781+0000","last_active":"2024-05-09T05:01:06.552957+0000","last_peered":"2024-05-09T05:01:06.552957+0000","last_clean":"2024-05-09T05:01:06.552957+0000","last_became_active":"2024-05-09T05:01:05.348507+0000","last_became_peered":"2024-05-09T05:01:05.348507+0000","last_unstale":"2024-05-09T05:01:06.552957+0000","last_undegraded":"2024-05-09T05:01:06.552957+0000","last_fullsized":"2024-05-09T05:01:06.552957+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:54:27.673057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.347746+0000","last_change":"2024-05-09T05:01:05.356233+0000","last_active":"2024-05-09T05:01:06.347746+0000","last_peered":"2024-05-09T05:01:06.347746+0000","last_clean":"2024-05-09T05:01:06.347746+0000","last_became_active":"2024-05-09T05:01:05.355862+0000","last_became_peered":"2024-05-09T05:01:05.355862+0000","last_unstale":"2024-05-09T05:01:06.347746+0000","last_undegraded":"2024-05-09T05:01:06.347746+0000","last_fullsized":"2024-05-09T05:01:06.347746+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:59:59.081918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.347626+0000","last_change":"2024-05-09T05:01:05.355637+0000","last_active":"2024-05-09T05:01:06.347626+0000","last_peered":"2024-05-09T05:01:06.347626+0000","last_clean":"2024-05-09T05:01:06.347626+0000","last_became_active":"2024-05-09T05:01:05.353042+0000","last_became_peered":"2024-05-09T05:01:05.353042+0000","last_unstale":"2024-05-09T05:01:06.347626+0000","last_undegraded":"2024-05-09T05:01:06.347626+0000","last_fullsized":"2024-05-09T05:01:06.347626+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:35.818258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.390033+0000","last_change":"2024-05-09T05:01:05.359894+0000","last_active":"2024-05-09T05:01:06.390033+0000","last_peered":"2024-05-09T05:01:06.390033+0000","last_clean":"2024-05-09T05:01:06.390033+0000","last_became_active":"2024-05-09T05:01:05.359626+0000","last_became_peered":"2024-05-09T05:01:05.359626+0000","last_unstale":"2024-05-09T05:01:06.390033+0000","last_undegraded":"2024-05-09T05:01:06.390033+0000","last_fullsized":"2024-05-09T05:01:06.390033+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:36:28.104239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.349627+0000","last_change":"2024-05-09T05:01:05.359081+0000","last_active":"2024-05-09T05:01:06.349627+0000","last_peered":"2024-05-09T05:01:06.349627+0000","last_clean":"2024-05-09T05:01:06.349627+0000","last_became_active":"2024-05-09T05:01:05.358508+0000","last_became_peered":"2024-05-09T05:01:05.358508+0000","last_unstale":"2024-05-09T05:01:06.349627+0000","last_undegraded":"2024-05-09T05:01:06.349627+0000","last_fullsized":"2024-05-09T05:01:06.349627+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:18:38.102212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344299+0000","last_change":"2024-05-09T05:01:05.344943+0000","last_active":"2024-05-09T05:01:06.344299+0000","last_peered":"2024-05-09T05:01:06.344299+0000","last_clean":"2024-05-09T05:01:06.344299+0000","last_became_active":"2024-05-09T05:01:05.344076+0000","last_became_peered":"2024-05-09T05:01:05.344076+0000","last_unstale":"2024-05-09T05:01:06.344299+0000","last_undegraded":"2024-05-09T05:01:06.344299+0000","last_fullsized":"2024-05-09T05:01:06.344299+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:04.027664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553090+0000","last_change":"2024-05-09T05:01:05.343323+0000","last_active":"2024-05-09T05:01:06.553090+0000","last_peered":"2024-05-09T05:01:06.553090+0000","last_clean":"2024-05-09T05:01:06.553090+0000","last_became_active":"2024-05-09T05:01:05.343026+0000","last_became_peered":"2024-05-09T05:01:05.343026+0000","last_unstale":"2024-05-09T05:01:06.553090+0000","last_undegraded":"2024-05-09T05:01:06.553090+0000","last_fullsized":"2024-05-09T05:01:06.553090+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:07:00.101790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.350027+0000","last_change":"2024-05-09T05:01:05.347643+0000","last_active":"2024-05-09T05:01:06.350027+0000","last_peered":"2024-05-09T05:01:06.350027+0000","last_clean":"2024-05-09T05:01:06.350027+0000","last_became_active":"2024-05-09T05:01:05.347205+0000","last_became_peered":"2024-05-09T05:01:05.347205+0000","last_unstale":"2024-05-09T05:01:06.350027+0000","last_undegraded":"2024-05-09T05:01:06.350027+0000","last_fullsized":"2024-05-09T05:01:06.350027+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:18:04.101345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.350029+0000","last_change":"2024-05-09T05:01:05.354158+0000","last_active":"2024-05-09T05:01:06.350029+0000","last_peered":"2024-05-09T05:01:06.350029+0000","last_clean":"2024-05-09T05:01:06.350029+0000","last_became_active":"2024-05-09T05:01:05.353169+0000","last_became_peered":"2024-05-09T05:01:05.353169+0000","last_unstale":"2024-05-09T05:01:06.350029+0000","last_undegraded":"2024-05-09T05:01:06.350029+0000","last_fullsized":"2024-05-09T05:01:06.350029+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:42:57.749068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.552937+0000","last_change":"2024-05-09T05:01:05.354354+0000","last_active":"2024-05-09T05:01:06.552937+0000","last_peered":"2024-05-09T05:01:06.552937+0000","last_clean":"2024-05-09T05:01:06.552937+0000","last_became_active":"2024-05-09T05:01:05.354075+0000","last_became_peered":"2024-05-09T05:01:05.354075+0000","last_unstale":"2024-05-09T05:01:06.552937+0000","last_undegraded":"2024-05-09T05:01:06.552937+0000","last_fullsized":"2024-05-09T05:01:06.552937+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:39:57.651574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.349732+0000","last_change":"2024-05-09T05:01:05.346339+0000","last_active":"2024-05-09T05:01:06.349732+0000","last_peered":"2024-05-09T05:01:06.349732+0000","last_clean":"2024-05-09T05:01:06.349732+0000","last_became_active":"2024-05-09T05:01:05.345971+0000","last_became_peered":"2024-05-09T05:01:05.345971+0000","last_unstale":"2024-05-09T05:01:06.349732+0000","last_undegraded":"2024-05-09T05:01:06.349732+0000","last_fullsized":"2024-05-09T05:01:06.349732+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:22:03.591207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553527+0000","last_change":"2024-05-09T05:01:05.352479+0000","last_active":"2024-05-09T05:01:06.553527+0000","last_peered":"2024-05-09T05:01:06.553527+0000","last_clean":"2024-05-09T05:01:06.553527+0000","last_became_active":"2024-05-09T05:01:05.352215+0000","last_became_peered":"2024-05-09T05:01:05.352215+0000","last_unstale":"2024-05-09T05:01:06.553527+0000","last_undegraded":"2024-05-09T05:01:06.553527+0000","last_fullsized":"2024-05-09T05:01:06.553527+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:20.893689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.349681+0000","last_change":"2024-05-09T05:01:05.344664+0000","last_active":"2024-05-09T05:01:06.349681+0000","last_peered":"2024-05-09T05:01:06.349681+0000","last_clean":"2024-05-09T05:01:06.349681+0000","last_became_active":"2024-05-09T05:01:05.344191+0000","last_became_peered":"2024-05-09T05:01:05.344191+0000","last_unstale":"2024-05-09T05:01:06.349681+0000","last_undegraded":"2024-05-09T05:01:06.349681+0000","last_fullsized":"2024-05-09T05:01:06.349681+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:06:27.722335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.345788+0000","last_change":"2024-05-09T05:01:05.351017+0000","last_active":"2024-05-09T05:01:06.345788+0000","last_peered":"2024-05-09T05:01:06.345788+0000","last_clean":"2024-05-09T05:01:06.345788+0000","last_became_active":"2024-05-09T05:01:05.350699+0000","last_became_peered":"2024-05-09T05:01:05.350699+0000","last_unstale":"2024-05-09T05:01:06.345788+0000","last_undegraded":"2024-05-09T05:01:06.345788+0000","last_fullsized":"2024-05-09T05:01:06.345788+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:52.250650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346518+0000","last_change":"2024-05-09T05:01:05.351131+0000","last_active":"2024-05-09T05:01:06.346518+0000","last_peered":"2024-05-09T05:01:06.346518+0000","last_clean":"2024-05-09T05:01:06.346518+0000","last_became_active":"2024-05-09T05:01:05.350831+0000","last_became_peered":"2024-05-09T05:01:05.350831+0000","last_unstale":"2024-05-09T05:01:06.346518+0000","last_undegraded":"2024-05-09T05:01:06.346518+0000","last_fullsized":"2024-05-09T05:01:06.346518+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:43:59.088798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553530+0000","last_change":"2024-05-09T05:01:05.353968+0000","last_active":"2024-05-09T05:01:06.553530+0000","last_peered":"2024-05-09T05:01:06.553530+0000","last_clean":"2024-05-09T05:01:06.553530+0000","last_became_active":"2024-05-09T05:01:05.353690+0000","last_became_peered":"2024-05-09T05:01:05.353690+0000","last_unstale":"2024-05-09T05:01:06.553530+0000","last_undegraded":"2024-05-09T05:01:06.553530+0000","last_fullsized":"2024-05-09T05:01:06.553530+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:11.997036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.347847+0000","last_change":"2024-05-09T05:01:05.349945+0000","last_active":"2024-05-09T05:01:06.347847+0000","last_peered":"2024-05-09T05:01:06.347847+0000","last_clean":"2024-05-09T05:01:06.347847+0000","last_became_active":"2024-05-09T05:01:05.349644+0000","last_became_peered":"2024-05-09T05:01:05.349644+0000","last_unstale":"2024-05-09T05:01:06.347847+0000","last_undegraded":"2024-05-09T05:01:06.347847+0000","last_fullsized":"2024-05-09T05:01:06.347847+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:29.806811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553710+0000","last_change":"2024-05-09T05:01:05.353538+0000","last_active":"2024-05-09T05:01:06.553710+0000","last_peered":"2024-05-09T05:01:06.553710+0000","last_clean":"2024-05-09T05:01:06.553710+0000","last_became_active":"2024-05-09T05:01:05.353216+0000","last_became_peered":"2024-05-09T05:01:05.353216+0000","last_unstale":"2024-05-09T05:01:06.553710+0000","last_undegraded":"2024-05-09T05:01:06.553710+0000","last_fullsized":"2024-05-09T05:01:06.553710+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:38.953101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.349803+0000","last_change":"2024-05-09T05:01:05.342529+0000","last_active":"2024-05-09T05:01:06.349803+0000","last_peered":"2024-05-09T05:01:06.349803+0000","last_clean":"2024-05-09T05:01:06.349803+0000","last_became_active":"2024-05-09T05:01:05.342230+0000","last_became_peered":"2024-05-09T05:01:05.342230+0000","last_unstale":"2024-05-09T05:01:06.349803+0000","last_undegraded":"2024-05-09T05:01:06.349803+0000","last_fullsized":"2024-05-09T05:01:06.349803+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:51:54.638779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553701+0000","last_change":"2024-05-09T05:01:05.353147+0000","last_active":"2024-05-09T05:01:06.553701+0000","last_peered":"2024-05-09T05:01:06.553701+0000","last_clean":"2024-05-09T05:01:06.553701+0000","last_became_active":"2024-05-09T05:01:05.352845+0000","last_became_peered":"2024-05-09T05:01:05.352845+0000","last_unstale":"2024-05-09T05:01:06.553701+0000","last_undegraded":"2024-05-09T05:01:06.553701+0000","last_fullsized":"2024-05-09T05:01:06.553701+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:38.953101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.345816+0000","last_change":"2024-05-09T05:01:05.353291+0000","last_active":"2024-05-09T05:01:06.345816+0000","last_peered":"2024-05-09T05:01:06.345816+0000","last_clean":"2024-05-09T05:01:06.345816+0000","last_became_active":"2024-05-09T05:01:05.353024+0000","last_became_peered":"2024-05-09T05:01:05.353024+0000","last_unstale":"2024-05-09T05:01:06.345816+0000","last_undegraded":"2024-05-09T05:01:06.345816+0000","last_fullsized":"2024-05-09T05:01:06.345816+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:07:00.101790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.347489+0000","last_change":"2024-05-09T05:01:05.353091+0000","last_active":"2024-05-09T05:01:06.347489+0000","last_peered":"2024-05-09T05:01:06.347489+0000","last_clean":"2024-05-09T05:01:06.347489+0000","last_became_active":"2024-05-09T05:01:05.352787+0000","last_became_peered":"2024-05-09T05:01:05.352787+0000","last_unstale":"2024-05-09T05:01:06.347489+0000","last_undegraded":"2024-05-09T05:01:06.347489+0000","last_fullsized":"2024-05-09T05:01:06.347489+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:12:42.318279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553952+0000","last_change":"2024-05-09T05:01:05.348707+0000","last_active":"2024-05-09T05:01:06.553952+0000","last_peered":"2024-05-09T05:01:06.553952+0000","last_clean":"2024-05-09T05:01:06.553952+0000","last_became_active":"2024-05-09T05:01:05.348447+0000","last_became_peered":"2024-05-09T05:01:05.348447+0000","last_unstale":"2024-05-09T05:01:06.553952+0000","last_undegraded":"2024-05-09T05:01:06.553952+0000","last_fullsized":"2024-05-09T05:01:06.553952+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:06:51.148491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.390285+0000","last_change":"2024-05-09T05:01:05.349074+0000","last_active":"2024-05-09T05:01:06.390285+0000","last_peered":"2024-05-09T05:01:06.390285+0000","last_clean":"2024-05-09T05:01:06.390285+0000","last_became_active":"2024-05-09T05:01:05.348803+0000","last_became_peered":"2024-05-09T05:01:05.348803+0000","last_unstale":"2024-05-09T05:01:06.390285+0000","last_undegraded":"2024-05-09T05:01:06.390285+0000","last_fullsized":"2024-05-09T05:01:06.390285+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:39:57.651574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553728+0000","last_change":"2024-05-09T05:01:05.349504+0000","last_active":"2024-05-09T05:01:06.553728+0000","last_peered":"2024-05-09T05:01:06.553728+0000","last_clean":"2024-05-09T05:01:06.553728+0000","last_became_active":"2024-05-09T05:01:05.349187+0000","last_became_peered":"2024-05-09T05:01:05.349187+0000","last_unstale":"2024-05-09T05:01:06.553728+0000","last_undegraded":"2024-05-09T05:01:06.553728+0000","last_fullsized":"2024-05-09T05:01:06.553728+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:56:54.390891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.349962+0000","last_change":"2024-05-09T05:01:05.354923+0000","last_active":"2024-05-09T05:01:06.349962+0000","last_peered":"2024-05-09T05:01:06.349962+0000","last_clean":"2024-05-09T05:01:06.349962+0000","last_became_active":"2024-05-09T05:01:05.354530+0000","last_became_peered":"2024-05-09T05:01:05.354530+0000","last_unstale":"2024-05-09T05:01:06.349962+0000","last_undegraded":"2024-05-09T05:01:06.349962+0000","last_fullsized":"2024-05-09T05:01:06.349962+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:27.412828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553338+0000","last_change":"2024-05-09T05:01:05.349745+0000","last_active":"2024-05-09T05:01:06.553338+0000","last_peered":"2024-05-09T05:01:06.553338+0000","last_clean":"2024-05-09T05:01:06.553338+0000","last_became_active":"2024-05-09T05:01:05.349309+0000","last_became_peered":"2024-05-09T05:01:05.349309+0000","last_unstale":"2024-05-09T05:01:06.553338+0000","last_undegraded":"2024-05-09T05:01:06.553338+0000","last_fullsized":"2024-05-09T05:01:06.553338+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:52:45.792920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553383+0000","last_change":"2024-05-09T05:01:05.352247+0000","last_active":"2024-05-09T05:01:06.553383+0000","last_peered":"2024-05-09T05:01:06.553383+0000","last_clean":"2024-05-09T05:01:06.553383+0000","last_became_active":"2024-05-09T05:01:05.351968+0000","last_became_peered":"2024-05-09T05:01:05.351968+0000","last_unstale":"2024-05-09T05:01:06.553383+0000","last_undegraded":"2024-05-09T05:01:06.553383+0000","last_fullsized":"2024-05-09T05:01:06.553383+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:20.893689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346106+0000","last_change":"2024-05-09T05:01:05.353823+0000","last_active":"2024-05-09T05:01:06.346106+0000","last_peered":"2024-05-09T05:01:06.346106+0000","last_clean":"2024-05-09T05:01:06.346106+0000","last_became_active":"2024-05-09T05:01:05.353577+0000","last_became_peered":"2024-05-09T05:01:05.353577+0000","last_unstale":"2024-05-09T05:01:06.346106+0000","last_undegraded":"2024-05-09T05:01:06.346106+0000","last_fullsized":"2024-05-09T05:01:06.346106+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:04.027664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.389928+0000","last_change":"2024-05-09T05:01:05.352590+0000","last_active":"2024-05-09T05:01:06.389928+0000","last_peered":"2024-05-09T05:01:06.389928+0000","last_clean":"2024-05-09T05:01:06.389928+0000","last_became_active":"2024-05-09T05:01:05.352330+0000","last_became_peered":"2024-05-09T05:01:05.352330+0000","last_unstale":"2024-05-09T05:01:06.389928+0000","last_undegraded":"2024-05-09T05:01:06.389928+0000","last_fullsized":"2024-05-09T05:01:06.389928+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:04:43.523719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346433+0000","last_change":"2024-05-09T05:01:05.350101+0000","last_active":"2024-05-09T05:01:06.346433+0000","last_peered":"2024-05-09T05:01:06.346433+0000","last_clean":"2024-05-09T05:01:06.346433+0000","last_became_active":"2024-05-09T05:01:05.349831+0000","last_became_peered":"2024-05-09T05:01:05.349831+0000","last_unstale":"2024-05-09T05:01:06.346433+0000","last_undegraded":"2024-05-09T05:01:06.346433+0000","last_fullsized":"2024-05-09T05:01:06.346433+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:22:03.591207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"35'4","reported_seq":18,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.554023+0000","last_change":"2024-05-09T05:01:04.339653+0000","last_active":"2024-05-09T05:01:06.554023+0000","last_peered":"2024-05-09T05:01:06.554023+0000","last_clean":"2024-05-09T05:01:06.554023+0000","last_became_active":"2024-05-09T05:01:04.339376+0000","last_became_peered":"2024-05-09T05:01:04.339376+0000","last_unstale":"2024-05-09T05:01:06.554023+0000","last_undegraded":"2024-05-09T05:01:06.554023+0000","last_fullsized":"2024-05-09T05:01:06.554023+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:08.325696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.390162+0000","last_change":"2024-05-09T05:01:05.349287+0000","last_active":"2024-05-09T05:01:06.390162+0000","last_peered":"2024-05-09T05:01:06.390162+0000","last_clean":"2024-05-09T05:01:06.390162+0000","last_became_active":"2024-05-09T05:01:05.349017+0000","last_became_peered":"2024-05-09T05:01:05.349017+0000","last_unstale":"2024-05-09T05:01:06.390162+0000","last_undegraded":"2024-05-09T05:01:06.390162+0000","last_fullsized":"2024-05-09T05:01:06.390162+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:00:27.078945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"35'1","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553031+0000","last_change":"2024-05-09T05:01:04.336251+0000","last_active":"2024-05-09T05:01:06.553031+0000","last_peered":"2024-05-09T05:01:06.553031+0000","last_clean":"2024-05-09T05:01:06.553031+0000","last_became_active":"2024-05-09T05:01:04.335947+0000","last_became_peered":"2024-05-09T05:01:04.335947+0000","last_unstale":"2024-05-09T05:01:06.553031+0000","last_undegraded":"2024-05-09T05:01:06.553031+0000","last_fullsized":"2024-05-09T05:01:06.553031+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:27.200081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.554029+0000","last_change":"2024-05-09T05:01:05.357262+0000","last_active":"2024-05-09T05:01:06.554029+0000","last_peered":"2024-05-09T05:01:06.554029+0000","last_clean":"2024-05-09T05:01:06.554029+0000","last_became_active":"2024-05-09T05:01:05.356588+0000","last_became_peered":"2024-05-09T05:01:05.356588+0000","last_unstale":"2024-05-09T05:01:06.554029+0000","last_undegraded":"2024-05-09T05:01:06.554029+0000","last_fullsized":"2024-05-09T05:01:06.554029+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:04:43.523719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"35'2","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.390188+0000","last_change":"2024-05-09T05:01:04.339576+0000","last_active":"2024-05-09T05:01:06.390188+0000","last_peered":"2024-05-09T05:01:06.390188+0000","last_clean":"2024-05-09T05:01:06.390188+0000","last_became_active":"2024-05-09T05:01:04.339314+0000","last_became_peered":"2024-05-09T05:01:04.339314+0000","last_unstale":"2024-05-09T05:01:06.390188+0000","last_undegraded":"2024-05-09T05:01:06.390188+0000","last_fullsized":"2024-05-09T05:01:06.390188+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:34:23.217355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344791+0000","last_change":"2024-05-09T05:01:05.351578+0000","last_active":"2024-05-09T05:01:06.344791+0000","last_peered":"2024-05-09T05:01:06.344791+0000","last_clean":"2024-05-09T05:01:06.344791+0000","last_became_active":"2024-05-09T05:01:05.351300+0000","last_became_peered":"2024-05-09T05:01:05.351300+0000","last_unstale":"2024-05-09T05:01:06.344791+0000","last_undegraded":"2024-05-09T05:01:06.344791+0000","last_fullsized":"2024-05-09T05:01:06.344791+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:42:57.749068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"35'2","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553777+0000","last_change":"2024-05-09T05:01:04.340150+0000","last_active":"2024-05-09T05:01:06.553777+0000","last_peered":"2024-05-09T05:01:06.553777+0000","last_clean":"2024-05-09T05:01:06.553777+0000","last_became_active":"2024-05-09T05:01:04.339879+0000","last_became_peered":"2024-05-09T05:01:04.339879+0000","last_unstale":"2024-05-09T05:01:06.553777+0000","last_undegraded":"2024-05-09T05:01:06.553777+0000","last_fullsized":"2024-05-09T05:01:06.553777+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:14:27.134789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.552743+0000","last_change":"2024-05-09T05:01:05.348052+0000","last_active":"2024-05-09T05:01:06.552743+0000","last_peered":"2024-05-09T05:01:06.552743+0000","last_clean":"2024-05-09T05:01:06.552743+0000","last_became_active":"2024-05-09T05:01:05.347720+0000","last_became_peered":"2024-05-09T05:01:05.347720+0000","last_unstale":"2024-05-09T05:01:06.552743+0000","last_undegraded":"2024-05-09T05:01:06.552743+0000","last_fullsized":"2024-05-09T05:01:06.552743+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:43:40.752597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"35'1","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.349853+0000","last_change":"2024-05-09T05:01:04.340877+0000","last_active":"2024-05-09T05:01:06.349853+0000","last_peered":"2024-05-09T05:01:06.349853+0000","last_clean":"2024-05-09T05:01:06.349853+0000","last_became_active":"2024-05-09T05:01:04.340598+0000","last_became_peered":"2024-05-09T05:01:04.340598+0000","last_unstale":"2024-05-09T05:01:06.349853+0000","last_undegraded":"2024-05-09T05:01:06.349853+0000","last_fullsized":"2024-05-09T05:01:06.349853+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:56:53.386203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.349812+0000","last_change":"2024-05-09T05:01:05.351056+0000","last_active":"2024-05-09T05:01:06.349812+0000","last_peered":"2024-05-09T05:01:06.349812+0000","last_clean":"2024-05-09T05:01:06.349812+0000","last_became_active":"2024-05-09T05:01:05.350784+0000","last_became_peered":"2024-05-09T05:01:05.350784+0000","last_unstale":"2024-05-09T05:01:06.349812+0000","last_undegraded":"2024-05-09T05:01:06.349812+0000","last_fullsized":"2024-05-09T05:01:06.349812+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:43:59.088798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"35'1","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.345944+0000","last_change":"2024-05-09T05:01:04.341057+0000","last_active":"2024-05-09T05:01:06.345944+0000","last_peered":"2024-05-09T05:01:06.345944+0000","last_clean":"2024-05-09T05:01:06.345944+0000","last_became_active":"2024-05-09T05:01:04.340771+0000","last_became_peered":"2024-05-09T05:01:04.340771+0000","last_unstale":"2024-05-09T05:01:06.345944+0000","last_undegraded":"2024-05-09T05:01:06.345944+0000","last_fullsized":"2024-05-09T05:01:06.345944+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:02:59.742688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553953+0000","last_change":"2024-05-09T05:01:05.354344+0000","last_active":"2024-05-09T05:01:06.553953+0000","last_peered":"2024-05-09T05:01:06.553953+0000","last_clean":"2024-05-09T05:01:06.553953+0000","last_became_active":"2024-05-09T05:01:05.354039+0000","last_became_peered":"2024-05-09T05:01:05.354039+0000","last_unstale":"2024-05-09T05:01:06.553953+0000","last_undegraded":"2024-05-09T05:01:06.553953+0000","last_fullsized":"2024-05-09T05:01:06.553953+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:06:51.148491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.390002+0000","last_change":"2024-05-09T05:01:04.338831+0000","last_active":"2024-05-09T05:01:06.390002+0000","last_peered":"2024-05-09T05:01:06.390002+0000","last_clean":"2024-05-09T05:01:06.390002+0000","last_became_active":"2024-05-09T05:01:04.338565+0000","last_became_peered":"2024-05-09T05:01:04.338565+0000","last_unstale":"2024-05-09T05:01:06.390002+0000","last_undegraded":"2024-05-09T05:01:06.390002+0000","last_fullsized":"2024-05-09T05:01:06.390002+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:04.648238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344468+0000","last_change":"2024-05-09T05:01:05.347212+0000","last_active":"2024-05-09T05:01:06.344468+0000","last_peered":"2024-05-09T05:01:06.344468+0000","last_clean":"2024-05-09T05:01:06.344468+0000","last_became_active":"2024-05-09T05:01:05.346912+0000","last_became_peered":"2024-05-09T05:01:05.346912+0000","last_unstale":"2024-05-09T05:01:06.344468+0000","last_undegraded":"2024-05-09T05:01:06.344468+0000","last_fullsized":"2024-05-09T05:01:06.344468+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:06:27.722335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553379+0000","last_change":"2024-05-09T05:01:04.339770+0000","last_active":"2024-05-09T05:01:06.553379+0000","last_peered":"2024-05-09T05:01:06.553379+0000","last_clean":"2024-05-09T05:01:06.553379+0000","last_became_active":"2024-05-09T05:01:04.339497+0000","last_became_peered":"2024-05-09T05:01:04.339497+0000","last_unstale":"2024-05-09T05:01:06.553379+0000","last_undegraded":"2024-05-09T05:01:06.553379+0000","last_fullsized":"2024-05-09T05:01:06.553379+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:32:59.160180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.347878+0000","last_change":"2024-05-09T05:01:05.357090+0000","last_active":"2024-05-09T05:01:06.347878+0000","last_peered":"2024-05-09T05:01:06.347878+0000","last_clean":"2024-05-09T05:01:06.347878+0000","last_became_active":"2024-05-09T05:01:05.356584+0000","last_became_peered":"2024-05-09T05:01:05.356584+0000","last_unstale":"2024-05-09T05:01:06.347878+0000","last_undegraded":"2024-05-09T05:01:06.347878+0000","last_fullsized":"2024-05-09T05:01:06.347878+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:38:35.684279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344790+0000","last_change":"2024-05-09T05:01:04.341197+0000","last_active":"2024-05-09T05:01:06.344790+0000","last_peered":"2024-05-09T05:01:06.344790+0000","last_clean":"2024-05-09T05:01:06.344790+0000","last_became_active":"2024-05-09T05:01:04.340797+0000","last_became_peered":"2024-05-09T05:01:04.340797+0000","last_unstale":"2024-05-09T05:01:06.344790+0000","last_undegraded":"2024-05-09T05:01:06.344790+0000","last_fullsized":"2024-05-09T05:01:06.344790+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:56:53.386203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346282+0000","last_change":"2024-05-09T05:01:05.347902+0000","last_active":"2024-05-09T05:01:06.346282+0000","last_peered":"2024-05-09T05:01:06.346282+0000","last_clean":"2024-05-09T05:01:06.346282+0000","last_became_active":"2024-05-09T05:01:05.347629+0000","last_became_peered":"2024-05-09T05:01:05.347629+0000","last_unstale":"2024-05-09T05:01:06.346282+0000","last_undegraded":"2024-05-09T05:01:06.346282+0000","last_fullsized":"2024-05-09T05:01:06.346282+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:58.791766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.350132+0000","last_change":"2024-05-09T05:01:04.341543+0000","last_active":"2024-05-09T05:01:06.350132+0000","last_peered":"2024-05-09T05:01:06.350132+0000","last_clean":"2024-05-09T05:01:06.350132+0000","last_became_active":"2024-05-09T05:01:04.341237+0000","last_became_peered":"2024-05-09T05:01:04.341237+0000","last_unstale":"2024-05-09T05:01:06.350132+0000","last_undegraded":"2024-05-09T05:01:06.350132+0000","last_fullsized":"2024-05-09T05:01:06.350132+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:37.948413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.552997+0000","last_change":"2024-05-09T05:01:05.347840+0000","last_active":"2024-05-09T05:01:06.552997+0000","last_peered":"2024-05-09T05:01:06.552997+0000","last_clean":"2024-05-09T05:01:06.552997+0000","last_became_active":"2024-05-09T05:01:05.347382+0000","last_became_peered":"2024-05-09T05:01:05.347382+0000","last_unstale":"2024-05-09T05:01:06.552997+0000","last_undegraded":"2024-05-09T05:01:06.552997+0000","last_fullsized":"2024-05-09T05:01:06.552997+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:26:52.250650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"35'3","reported_seq":17,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.350146+0000","last_change":"2024-05-09T05:01:04.337652+0000","last_active":"2024-05-09T05:01:06.350146+0000","last_peered":"2024-05-09T05:01:06.350146+0000","last_clean":"2024-05-09T05:01:06.350146+0000","last_became_active":"2024-05-09T05:01:04.337383+0000","last_became_peered":"2024-05-09T05:01:04.337383+0000","last_unstale":"2024-05-09T05:01:06.350146+0000","last_undegraded":"2024-05-09T05:01:06.350146+0000","last_fullsized":"2024-05-09T05:01:06.350146+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:10.992348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.554100+0000","last_change":"2024-05-09T05:01:05.355083+0000","last_active":"2024-05-09T05:01:06.554100+0000","last_peered":"2024-05-09T05:01:06.554100+0000","last_clean":"2024-05-09T05:01:06.554100+0000","last_became_active":"2024-05-09T05:01:05.354816+0000","last_became_peered":"2024-05-09T05:01:05.354816+0000","last_unstale":"2024-05-09T05:01:06.554100+0000","last_undegraded":"2024-05-09T05:01:06.554100+0000","last_fullsized":"2024-05-09T05:01:06.554100+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:43:40.752597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"35'6","reported_seq":20,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344508+0000","last_change":"2024-05-09T05:01:04.335947+0000","last_active":"2024-05-09T05:01:06.344508+0000","last_peered":"2024-05-09T05:01:06.344508+0000","last_clean":"2024-05-09T05:01:06.344508+0000","last_became_active":"2024-05-09T05:01:04.335678+0000","last_became_peered":"2024-05-09T05:01:04.335678+0000","last_unstale":"2024-05-09T05:01:06.344508+0000","last_undegraded":"2024-05-09T05:01:06.344508+0000","last_fullsized":"2024-05-09T05:01:06.344508+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:42:37.948413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.552797+0000","last_change":"2024-05-09T05:01:05.349002+0000","last_active":"2024-05-09T05:01:06.552797+0000","last_peered":"2024-05-09T05:01:06.552797+0000","last_clean":"2024-05-09T05:01:06.552797+0000","last_became_active":"2024-05-09T05:01:05.348746+0000","last_became_peered":"2024-05-09T05:01:05.348746+0000","last_unstale":"2024-05-09T05:01:06.552797+0000","last_undegraded":"2024-05-09T05:01:06.552797+0000","last_fullsized":"2024-05-09T05:01:06.552797+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:02:46.125356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"35'2","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344635+0000","last_change":"2024-05-09T05:01:04.344726+0000","last_active":"2024-05-09T05:01:06.344635+0000","last_peered":"2024-05-09T05:01:06.344635+0000","last_clean":"2024-05-09T05:01:06.344635+0000","last_became_active":"2024-05-09T05:01:04.344384+0000","last_became_peered":"2024-05-09T05:01:04.344384+0000","last_unstale":"2024-05-09T05:01:06.344635+0000","last_undegraded":"2024-05-09T05:01:06.344635+0000","last_fullsized":"2024-05-09T05:01:06.344635+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:06:50.143802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344460+0000","last_change":"2024-05-09T05:01:05.340042+0000","last_active":"2024-05-09T05:01:06.344460+0000","last_peered":"2024-05-09T05:01:06.344460+0000","last_clean":"2024-05-09T05:01:06.344460+0000","last_became_active":"2024-05-09T05:01:05.339742+0000","last_became_peered":"2024-05-09T05:01:05.339742+0000","last_unstale":"2024-05-09T05:01:06.344460+0000","last_undegraded":"2024-05-09T05:01:06.344460+0000","last_fullsized":"2024-05-09T05:01:06.344460+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:18:38.102212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.389909+0000","last_change":"2024-05-09T05:01:04.345746+0000","last_active":"2024-05-09T05:01:06.389909+0000","last_peered":"2024-05-09T05:01:06.389909+0000","last_clean":"2024-05-09T05:01:06.389909+0000","last_became_active":"2024-05-09T05:01:04.345514+0000","last_became_peered":"2024-05-09T05:01:04.345514+0000","last_unstale":"2024-05-09T05:01:06.389909+0000","last_undegraded":"2024-05-09T05:01:06.389909+0000","last_fullsized":"2024-05-09T05:01:06.389909+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:19.502728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346015+0000","last_change":"2024-05-09T05:01:05.353988+0000","last_active":"2024-05-09T05:01:06.346015+0000","last_peered":"2024-05-09T05:01:06.346015+0000","last_clean":"2024-05-09T05:01:06.346015+0000","last_became_active":"2024-05-09T05:01:05.353679+0000","last_became_peered":"2024-05-09T05:01:05.353679+0000","last_unstale":"2024-05-09T05:01:06.346015+0000","last_undegraded":"2024-05-09T05:01:06.346015+0000","last_fullsized":"2024-05-09T05:01:06.346015+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:45:56.221913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553160+0000","last_change":"2024-05-09T05:01:04.337903+0000","last_active":"2024-05-09T05:01:06.553160+0000","last_peered":"2024-05-09T05:01:06.553160+0000","last_clean":"2024-05-09T05:01:06.553160+0000","last_became_active":"2024-05-09T05:01:04.337616+0000","last_became_peered":"2024-05-09T05:01:04.337616+0000","last_unstale":"2024-05-09T05:01:06.553160+0000","last_undegraded":"2024-05-09T05:01:06.553160+0000","last_fullsized":"2024-05-09T05:01:06.553160+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:14:27.134789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344520+0000","last_change":"2024-05-09T05:01:05.340539+0000","last_active":"2024-05-09T05:01:06.344520+0000","last_peered":"2024-05-09T05:01:06.344520+0000","last_clean":"2024-05-09T05:01:06.344520+0000","last_became_active":"2024-05-09T05:01:05.340233+0000","last_became_peered":"2024-05-09T05:01:05.340233+0000","last_unstale":"2024-05-09T05:01:06.344520+0000","last_undegraded":"2024-05-09T05:01:06.344520+0000","last_fullsized":"2024-05-09T05:01:06.344520+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:58.791766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"35'8","reported_seq":22,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.345791+0000","last_change":"2024-05-09T05:01:04.338086+0000","last_active":"2024-05-09T05:01:06.345791+0000","last_peered":"2024-05-09T05:01:06.345791+0000","last_clean":"2024-05-09T05:01:06.345791+0000","last_became_active":"2024-05-09T05:01:04.337822+0000","last_became_peered":"2024-05-09T05:01:04.337822+0000","last_unstale":"2024-05-09T05:01:06.345791+0000","last_undegraded":"2024-05-09T05:01:06.345791+0000","last_fullsized":"2024-05-09T05:01:06.345791+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:34:23.217355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344529+0000","last_change":"2024-05-09T05:01:05.350366+0000","last_active":"2024-05-09T05:01:06.344529+0000","last_peered":"2024-05-09T05:01:06.344529+0000","last_clean":"2024-05-09T05:01:06.344529+0000","last_became_active":"2024-05-09T05:01:05.350068+0000","last_became_peered":"2024-05-09T05:01:05.350068+0000","last_unstale":"2024-05-09T05:01:06.344529+0000","last_undegraded":"2024-05-09T05:01:06.344529+0000","last_fullsized":"2024-05-09T05:01:06.344529+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:22:03.591207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"35'1","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.350089+0000","last_change":"2024-05-09T05:01:04.339855+0000","last_active":"2024-05-09T05:01:06.350089+0000","last_peered":"2024-05-09T05:01:06.350089+0000","last_clean":"2024-05-09T05:01:06.350089+0000","last_became_active":"2024-05-09T05:01:04.339601+0000","last_became_peered":"2024-05-09T05:01:04.339601+0000","last_unstale":"2024-05-09T05:01:06.350089+0000","last_undegraded":"2024-05-09T05:01:06.350089+0000","last_fullsized":"2024-05-09T05:01:06.350089+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:52:44.788232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.350081+0000","last_change":"2024-05-09T05:01:05.341841+0000","last_active":"2024-05-09T05:01:06.350081+0000","last_peered":"2024-05-09T05:01:06.350081+0000","last_clean":"2024-05-09T05:01:06.350081+0000","last_became_active":"2024-05-09T05:01:05.341643+0000","last_became_peered":"2024-05-09T05:01:05.341643+0000","last_unstale":"2024-05-09T05:01:06.350081+0000","last_undegraded":"2024-05-09T05:01:06.350081+0000","last_fullsized":"2024-05-09T05:01:06.350081+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:29:27.988279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"35'2","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.554067+0000","last_change":"2024-05-09T05:01:04.338411+0000","last_active":"2024-05-09T05:01:06.554067+0000","last_peered":"2024-05-09T05:01:06.554067+0000","last_clean":"2024-05-09T05:01:06.554067+0000","last_became_active":"2024-05-09T05:01:04.338123+0000","last_became_peered":"2024-05-09T05:01:04.338123+0000","last_unstale":"2024-05-09T05:01:06.554067+0000","last_undegraded":"2024-05-09T05:01:06.554067+0000","last_fullsized":"2024-05-09T05:01:06.554067+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:08.325696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553953+0000","last_change":"2024-05-09T05:01:05.356091+0000","last_active":"2024-05-09T05:01:06.553953+0000","last_peered":"2024-05-09T05:01:06.553953+0000","last_clean":"2024-05-09T05:01:06.553953+0000","last_became_active":"2024-05-09T05:01:05.355752+0000","last_became_peered":"2024-05-09T05:01:05.355752+0000","last_unstale":"2024-05-09T05:01:06.553953+0000","last_undegraded":"2024-05-09T05:01:06.553953+0000","last_fullsized":"2024-05-09T05:01:06.553953+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:56:54.390891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"35'1","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.347944+0000","last_change":"2024-05-09T05:01:04.336039+0000","last_active":"2024-05-09T05:01:06.347944+0000","last_peered":"2024-05-09T05:01:06.347944+0000","last_clean":"2024-05-09T05:01:06.347944+0000","last_became_active":"2024-05-09T05:01:04.335735+0000","last_became_peered":"2024-05-09T05:01:04.335735+0000","last_unstale":"2024-05-09T05:01:06.347944+0000","last_undegraded":"2024-05-09T05:01:06.347944+0000","last_fullsized":"2024-05-09T05:01:06.347944+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:58.290034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.348127+0000","last_change":"2024-05-09T05:01:05.358170+0000","last_active":"2024-05-09T05:01:06.348127+0000","last_peered":"2024-05-09T05:01:06.348127+0000","last_clean":"2024-05-09T05:01:06.348127+0000","last_became_active":"2024-05-09T05:01:05.357840+0000","last_became_peered":"2024-05-09T05:01:05.357840+0000","last_unstale":"2024-05-09T05:01:06.348127+0000","last_undegraded":"2024-05-09T05:01:06.348127+0000","last_fullsized":"2024-05-09T05:01:06.348127+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:33:00.164869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.390035+0000","last_change":"2024-05-09T05:01:04.340174+0000","last_active":"2024-05-09T05:01:06.390035+0000","last_peered":"2024-05-09T05:01:06.390035+0000","last_clean":"2024-05-09T05:01:06.390035+0000","last_became_active":"2024-05-09T05:01:04.339896+0000","last_became_peered":"2024-05-09T05:01:04.339896+0000","last_unstale":"2024-05-09T05:01:06.390035+0000","last_undegraded":"2024-05-09T05:01:06.390035+0000","last_fullsized":"2024-05-09T05:01:06.390035+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:27.200081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346219+0000","last_change":"2024-05-09T05:01:05.354792+0000","last_active":"2024-05-09T05:01:06.346219+0000","last_peered":"2024-05-09T05:01:06.346219+0000","last_clean":"2024-05-09T05:01:06.346219+0000","last_became_active":"2024-05-09T05:01:05.354395+0000","last_became_peered":"2024-05-09T05:01:05.354395+0000","last_unstale":"2024-05-09T05:01:06.346219+0000","last_undegraded":"2024-05-09T05:01:06.346219+0000","last_fullsized":"2024-05-09T05:01:06.346219+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:06:27.722335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"35'2","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.347533+0000","last_change":"2024-05-09T05:01:04.337165+0000","last_active":"2024-05-09T05:01:06.347533+0000","last_peered":"2024-05-09T05:01:06.347533+0000","last_clean":"2024-05-09T05:01:06.347533+0000","last_became_active":"2024-05-09T05:01:04.336861+0000","last_became_peered":"2024-05-09T05:01:04.336861+0000","last_unstale":"2024-05-09T05:01:06.347533+0000","last_undegraded":"2024-05-09T05:01:06.347533+0000","last_fullsized":"2024-05-09T05:01:06.347533+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:49.639516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.552803+0000","last_change":"2024-05-09T05:01:05.345491+0000","last_active":"2024-05-09T05:01:06.552803+0000","last_peered":"2024-05-09T05:01:06.552803+0000","last_clean":"2024-05-09T05:01:06.552803+0000","last_became_active":"2024-05-09T05:01:05.345155+0000","last_became_peered":"2024-05-09T05:01:05.345155+0000","last_unstale":"2024-05-09T05:01:06.552803+0000","last_undegraded":"2024-05-09T05:01:06.552803+0000","last_fullsized":"2024-05-09T05:01:06.552803+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:36:28.104239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.347515+0000","last_change":"2024-05-09T05:01:04.338378+0000","last_active":"2024-05-09T05:01:06.347515+0000","last_peered":"2024-05-09T05:01:06.347515+0000","last_clean":"2024-05-09T05:01:06.347515+0000","last_became_active":"2024-05-09T05:01:04.338002+0000","last_became_peered":"2024-05-09T05:01:04.338002+0000","last_unstale":"2024-05-09T05:01:06.347515+0000","last_undegraded":"2024-05-09T05:01:06.347515+0000","last_fullsized":"2024-05-09T05:01:06.347515+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:23:07.883786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.349723+0000","last_change":"2024-05-09T05:01:05.347833+0000","last_active":"2024-05-09T05:01:06.349723+0000","last_peered":"2024-05-09T05:01:06.349723+0000","last_clean":"2024-05-09T05:01:06.349723+0000","last_became_active":"2024-05-09T05:01:05.346828+0000","last_became_peered":"2024-05-09T05:01:05.346828+0000","last_unstale":"2024-05-09T05:01:06.349723+0000","last_undegraded":"2024-05-09T05:01:06.349723+0000","last_fullsized":"2024-05-09T05:01:06.349723+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:19:58.791766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346070+0000","last_change":"2024-05-09T05:01:04.339021+0000","last_active":"2024-05-09T05:01:06.346070+0000","last_peered":"2024-05-09T05:01:06.346070+0000","last_clean":"2024-05-09T05:01:06.346070+0000","last_became_active":"2024-05-09T05:01:04.338750+0000","last_became_peered":"2024-05-09T05:01:04.338750+0000","last_unstale":"2024-05-09T05:01:06.346070+0000","last_undegraded":"2024-05-09T05:01:06.346070+0000","last_fullsized":"2024-05-09T05:01:06.346070+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:52:44.788232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344651+0000","last_change":"2024-05-09T05:01:05.347210+0000","last_active":"2024-05-09T05:01:06.344651+0000","last_peered":"2024-05-09T05:01:06.344651+0000","last_clean":"2024-05-09T05:01:06.344651+0000","last_became_active":"2024-05-09T05:01:05.346891+0000","last_became_peered":"2024-05-09T05:01:05.346891+0000","last_unstale":"2024-05-09T05:01:06.344651+0000","last_undegraded":"2024-05-09T05:01:06.344651+0000","last_fullsized":"2024-05-09T05:01:06.344651+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:43:59.088798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.347543+0000","last_change":"2024-05-09T05:01:04.336987+0000","last_active":"2024-05-09T05:01:06.347543+0000","last_peered":"2024-05-09T05:01:06.347543+0000","last_clean":"2024-05-09T05:01:06.347543+0000","last_became_active":"2024-05-09T05:01:04.336585+0000","last_became_peered":"2024-05-09T05:01:04.336585+0000","last_unstale":"2024-05-09T05:01:06.347543+0000","last_undegraded":"2024-05-09T05:01:06.347543+0000","last_fullsized":"2024-05-09T05:01:06.347543+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:32:47.893689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.390017+0000","last_change":"2024-05-09T05:01:05.348685+0000","last_active":"2024-05-09T05:01:06.390017+0000","last_peered":"2024-05-09T05:01:06.390017+0000","last_clean":"2024-05-09T05:01:06.390017+0000","last_became_active":"2024-05-09T05:01:05.348374+0000","last_became_peered":"2024-05-09T05:01:05.348374+0000","last_unstale":"2024-05-09T05:01:06.390017+0000","last_undegraded":"2024-05-09T05:01:06.390017+0000","last_fullsized":"2024-05-09T05:01:06.390017+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:02:46.125356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"35'2","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.350165+0000","last_change":"2024-05-09T05:01:04.342282+0000","last_active":"2024-05-09T05:01:06.350165+0000","last_peered":"2024-05-09T05:01:06.350165+0000","last_clean":"2024-05-09T05:01:06.350165+0000","last_became_active":"2024-05-09T05:01:04.342025+0000","last_became_peered":"2024-05-09T05:01:04.342025+0000","last_unstale":"2024-05-09T05:01:06.350165+0000","last_undegraded":"2024-05-09T05:01:06.350165+0000","last_fullsized":"2024-05-09T05:01:06.350165+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:19.889000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346609+0000","last_change":"2024-05-09T05:01:05.348627+0000","last_active":"2024-05-09T05:01:06.346609+0000","last_peered":"2024-05-09T05:01:06.346609+0000","last_clean":"2024-05-09T05:01:06.346609+0000","last_became_active":"2024-05-09T05:01:05.348330+0000","last_became_peered":"2024-05-09T05:01:05.348330+0000","last_unstale":"2024-05-09T05:01:06.346609+0000","last_undegraded":"2024-05-09T05:01:06.346609+0000","last_fullsized":"2024-05-09T05:01:06.346609+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:51:54.638779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"36'1","reported_seq":15,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.349142+0000","last_change":"2024-05-09T05:01:04.337098+0000","last_active":"2024-05-09T05:01:06.349142+0000","last_peered":"2024-05-09T05:01:06.349142+0000","last_clean":"2024-05-09T05:01:06.349142+0000","last_became_active":"2024-05-09T05:01:04.336769+0000","last_became_peered":"2024-05-09T05:01:04.336769+0000","last_unstale":"2024-05-09T05:01:06.349142+0000","last_undegraded":"2024-05-09T05:01:06.349142+0000","last_fullsized":"2024-05-09T05:01:06.349142+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:47:49.674854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344675+0000","last_change":"2024-05-09T05:01:05.357533+0000","last_active":"2024-05-09T05:01:06.344675+0000","last_peered":"2024-05-09T05:01:06.344675+0000","last_clean":"2024-05-09T05:01:06.344675+0000","last_became_active":"2024-05-09T05:01:05.357200+0000","last_became_peered":"2024-05-09T05:01:05.357200+0000","last_unstale":"2024-05-09T05:01:06.344675+0000","last_undegraded":"2024-05-09T05:01:06.344675+0000","last_fullsized":"2024-05-09T05:01:06.344675+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:27.412828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553043+0000","last_change":"2024-05-09T05:01:04.334350+0000","last_active":"2024-05-09T05:01:06.553043+0000","last_peered":"2024-05-09T05:01:06.553043+0000","last_clean":"2024-05-09T05:01:06.553043+0000","last_became_active":"2024-05-09T05:01:04.334089+0000","last_became_peered":"2024-05-09T05:01:04.334089+0000","last_unstale":"2024-05-09T05:01:06.553043+0000","last_undegraded":"2024-05-09T05:01:06.553043+0000","last_fullsized":"2024-05-09T05:01:06.553043+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:04.648238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.552911+0000","last_change":"2024-05-09T05:01:05.351990+0000","last_active":"2024-05-09T05:01:06.552911+0000","last_peered":"2024-05-09T05:01:06.552911+0000","last_clean":"2024-05-09T05:01:06.552911+0000","last_became_active":"2024-05-09T05:01:05.351720+0000","last_became_peered":"2024-05-09T05:01:05.351720+0000","last_unstale":"2024-05-09T05:01:06.552911+0000","last_undegraded":"2024-05-09T05:01:06.552911+0000","last_fullsized":"2024-05-09T05:01:06.552911+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:00:27.078945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.552965+0000","last_change":"2024-05-09T05:01:04.342024+0000","last_active":"2024-05-09T05:01:06.552965+0000","last_peered":"2024-05-09T05:01:06.552965+0000","last_clean":"2024-05-09T05:01:06.552965+0000","last_became_active":"2024-05-09T05:01:04.341764+0000","last_became_peered":"2024-05-09T05:01:04.341764+0000","last_unstale":"2024-05-09T05:01:06.552965+0000","last_undegraded":"2024-05-09T05:01:06.552965+0000","last_fullsized":"2024-05-09T05:01:06.552965+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:19.502728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.389978+0000","last_change":"2024-05-09T05:01:05.346758+0000","last_active":"2024-05-09T05:01:06.389978+0000","last_peered":"2024-05-09T05:01:06.389978+0000","last_clean":"2024-05-09T05:01:06.389978+0000","last_became_active":"2024-05-09T05:01:05.346442+0000","last_became_peered":"2024-05-09T05:01:05.346442+0000","last_unstale":"2024-05-09T05:01:06.389978+0000","last_undegraded":"2024-05-09T05:01:06.389978+0000","last_fullsized":"2024-05-09T05:01:06.389978+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:43:40.752597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553656+0000","last_change":"2024-05-09T05:01:04.330105+0000","last_active":"2024-05-09T05:01:06.553656+0000","last_peered":"2024-05-09T05:01:06.553656+0000","last_clean":"2024-05-09T05:01:06.553656+0000","last_became_active":"2024-05-09T05:01:04.329841+0000","last_became_peered":"2024-05-09T05:01:04.329841+0000","last_unstale":"2024-05-09T05:01:06.553656+0000","last_undegraded":"2024-05-09T05:01:06.553656+0000","last_fullsized":"2024-05-09T05:01:06.553656+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:32:59.160180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553757+0000","last_change":"2024-05-09T05:01:05.358864+0000","last_active":"2024-05-09T05:01:06.553757+0000","last_peered":"2024-05-09T05:01:06.553757+0000","last_clean":"2024-05-09T05:01:06.553757+0000","last_became_active":"2024-05-09T05:01:05.358593+0000","last_became_peered":"2024-05-09T05:01:05.358593+0000","last_unstale":"2024-05-09T05:01:06.553757+0000","last_undegraded":"2024-05-09T05:01:06.553757+0000","last_fullsized":"2024-05-09T05:01:06.553757+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:11.997036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.553710+0000","last_change":"2024-05-09T05:01:04.340828+0000","last_active":"2024-05-09T05:01:06.553710+0000","last_peered":"2024-05-09T05:01:06.553710+0000","last_clean":"2024-05-09T05:01:06.553710+0000","last_became_active":"2024-05-09T05:01:04.340564+0000","last_became_peered":"2024-05-09T05:01:04.340564+0000","last_unstale":"2024-05-09T05:01:06.553710+0000","last_undegraded":"2024-05-09T05:01:06.553710+0000","last_fullsized":"2024-05-09T05:01:06.553710+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:38:34.679590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346195+0000","last_change":"2024-05-09T05:01:05.348824+0000","last_active":"2024-05-09T05:01:06.346195+0000","last_peered":"2024-05-09T05:01:06.346195+0000","last_clean":"2024-05-09T05:01:06.346195+0000","last_became_active":"2024-05-09T05:01:05.348552+0000","last_became_peered":"2024-05-09T05:01:05.348552+0000","last_unstale":"2024-05-09T05:01:06.346195+0000","last_undegraded":"2024-05-09T05:01:06.346195+0000","last_fullsized":"2024-05-09T05:01:06.346195+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:18:38.102212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.344179+0000","last_change":"2024-05-09T05:01:04.341097+0000","last_active":"2024-05-09T05:01:06.344179+0000","last_peered":"2024-05-09T05:01:06.344179+0000","last_clean":"2024-05-09T05:01:06.344179+0000","last_became_active":"2024-05-09T05:01:04.340782+0000","last_became_peered":"2024-05-09T05:01:04.340782+0000","last_unstale":"2024-05-09T05:01:06.344179+0000","last_undegraded":"2024-05-09T05:01:06.344179+0000","last_fullsized":"2024-05-09T05:01:06.344179+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:10.992348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.350065+0000","last_change":"2024-05-09T05:01:05.356576+0000","last_active":"2024-05-09T05:01:06.350065+0000","last_peered":"2024-05-09T05:01:06.350065+0000","last_clean":"2024-05-09T05:01:06.350065+0000","last_became_active":"2024-05-09T05:01:05.356274+0000","last_became_peered":"2024-05-09T05:01:05.356274+0000","last_unstale":"2024-05-09T05:01:06.350065+0000","last_undegraded":"2024-05-09T05:01:06.350065+0000","last_fullsized":"2024-05-09T05:01:06.350065+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:04.317836+0000","last_clean_scrub_stamp":"2024-05-09T05:01:04.317836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:28.747938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346341+0000","last_change":"2024-05-09T05:01:04.336368+0000","last_active":"2024-05-09T05:01:06.346341+0000","last_peered":"2024-05-09T05:01:06.346341+0000","last_clean":"2024-05-09T05:01:06.346341+0000","last_became_active":"2024-05-09T05:01:04.336092+0000","last_became_peered":"2024-05-09T05:01:04.336092+0000","last_unstale":"2024-05-09T05:01:06.346341+0000","last_undegraded":"2024-05-09T05:01:06.346341+0000","last_fullsized":"2024-05-09T05:01:06.346341+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:03.313148+0000","last_clean_scrub_stamp":"2024-05-09T05:01:03.313148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:35:19.889000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":139,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-09T05:01:06.346303+0000","last_change":"2024-05-09T05:00:08.667158+0000","last_active":"2024-05-09T05:01:06.346303+0000","last_peered":"2024-05-09T05:01:06.346303+0000","last_clean":"2024-05-09T05:01:06.346303+0000","last_became_active":"2024-05-09T05:00:08.666316+0000","last_became_peered":"2024-05-09T05:00:08.666316+0000","last_unstale":"2024-05-09T05:01:06.346303+0000","last_undegraded":"2024-05-09T05:01:06.346303+0000","last_fullsized":"2024-05-09T05:01:06.346303+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_clean_scrub_stamp":"2024-05-09T05:00:07.370119+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:20.234643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574862,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343828,"statfs":{"total":96636764160,"available":96608079872,"internally_reserved":0,"allocated":1011712,"data_stored":760669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574863,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":169201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574863,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":168731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":169223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574862,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":417792,"data_stored":170265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574862,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":417792,"data_stored":175381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":417792,"data_stored":175401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1011712,"data_stored":765713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:01:09.751 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-09T05:01:09.752 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-09T05:01:09.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.899+0000 7fe0bd4da640 1 Processor -- start 2024-05-09T05:01:09.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.899+0000 7fe0bd4da640 1 -- start start 2024-05-09T05:01:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.900+0000 7fe0bd4da640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe0b8145080 0x7fe0b8070150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.900+0000 7fe0bd4da640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe0b80d0390 0x7fe0b8072100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.900+0000 7fe0bd4da640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0b80d0fb0 con 0x7fe0b8070690 2024-05-09T05:01:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.900+0000 7fe0bd4da640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fe0b8070b10 con 0x7fe0b80d0390 2024-05-09T05:01:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.900+0000 7fe0bd4da640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fe0b8071c70 con 0x7fe0b8145080 2024-05-09T05:01:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.900+0000 7fe0b77fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe0b80d0390 0x7fe0b8072100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.900+0000 7fe0b77fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe0b80d0390 0x7fe0b8072100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59810/0 (socket says 172.21.15.60:59810) 2024-05-09T05:01:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.900+0000 7fe0b77fe640 1 -- 172.21.15.60:0/660472214 learned_addr learned my addr 172.21.15.60:0/660472214 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.900+0000 7fe0b6ffd640 1 --2- 172.21.15.60:0/660472214 >> v2:172.21.15.114:3300/0 conn(0x7fe0b8145080 0x7fe0b8070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.900+0000 7fe0b77fe640 1 -- 172.21.15.60:0/660472214 >> v2:172.21.15.114:3300/0 conn(0x7fe0b8145080 msgr2=0x7fe0b8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.900+0000 7fe0b77fe640 1 --2- 172.21.15.60:0/660472214 >> v2:172.21.15.114:3300/0 conn(0x7fe0b8145080 0x7fe0b8070150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.900+0000 7fe0b77fe640 1 -- 172.21.15.60:0/660472214 >> v1:172.21.15.114:6789/0 conn(0x7fe0b8070690 legacy=0x7fe0b80cfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:09.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.900+0000 7fe0b77fe640 1 -- 172.21.15.60:0/660472214 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b8071940 con 0x7fe0b80d0390 2024-05-09T05:01:09.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.901+0000 7fe0b77fe640 1 --2- 172.21.15.60:0/660472214 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0b80d0390 0x7fe0b8072100 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8075300 tx=0x7fe0b002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3997d09955a1973f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:09.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.901+0000 7fe0b67fc640 1 -- 172.21.15.60:0/660472214 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b0036070 con 0x7fe0b80d0390 2024-05-09T05:01:09.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.901+0000 7fe0b67fc640 1 -- 172.21.15.60:0/660472214 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0b0031600 con 0x7fe0b80d0390 2024-05-09T05:01:09.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.901+0000 7fe0b67fc640 1 -- 172.21.15.60:0/660472214 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b00319d0 con 0x7fe0b80d0390 2024-05-09T05:01:09.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.901+0000 7fe0bd4da640 1 -- 172.21.15.60:0/660472214 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0b80d0390 msgr2=0x7fe0b8072100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:09.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.901+0000 7fe0bd4da640 1 --2- 172.21.15.60:0/660472214 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0b80d0390 0x7fe0b8072100 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8075300 tx=0x7fe0b002a610 comp rx=0 tx=0).stop 2024-05-09T05:01:09.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.902+0000 7fe0bd4da640 1 -- 172.21.15.60:0/660472214 shutdown_connections 2024-05-09T05:01:09.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.902+0000 7fe0bd4da640 1 --2- 172.21.15.60:0/660472214 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0b80d0390 0x7fe0b8072100 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.902+0000 7fe0bd4da640 1 --2- 172.21.15.60:0/660472214 >> v2:172.21.15.114:3300/0 conn(0x7fe0b8145080 0x7fe0b8070150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.902+0000 7fe0bd4da640 1 -- 172.21.15.60:0/660472214 >> 172.21.15.60:0/660472214 conn(0x7fe0b80b0da0 msgr2=0x7fe0b80b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:09.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.902+0000 7fe0bd4da640 1 -- 172.21.15.60:0/660472214 shutdown_connections 2024-05-09T05:01:09.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.902+0000 7fe0bd4da640 1 -- 172.21.15.60:0/660472214 wait complete. 2024-05-09T05:01:09.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.903+0000 7fe0bd4da640 1 Processor -- start 2024-05-09T05:01:09.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.903+0000 7fe0bd4da640 1 -- start start 2024-05-09T05:01:09.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.903+0000 7fe0bd4da640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0b8070690 0x7fe0b815b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:09.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.903+0000 7fe0bd4da640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0b80d0390 0x7fe0b8169970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.903+0000 7fe0bd4da640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0b8145080 0x7fe0b815dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.903+0000 7fe0bd4da640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b80d0cd0 con 0x7fe0b80d0390 2024-05-09T05:01:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.903+0000 7fe0bd4da640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b80d09f0 con 0x7fe0b8145080 2024-05-09T05:01:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.903+0000 7fe0bd4da640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe0b8072640 con 0x7fe0b8070690 2024-05-09T05:01:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.904+0000 7fe0affff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0b80d0390 0x7fe0b8169970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.904+0000 7fe0affff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0b80d0390 0x7fe0b8169970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59820/0 (socket says 172.21.15.60:59820) 2024-05-09T05:01:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.904+0000 7fe0affff640 1 -- 172.21.15.60:0/3707108901 learned_addr learned my addr 172.21.15.60:0/3707108901 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.904+0000 7fe0b6ffd640 1 --2- 172.21.15.60:0/3707108901 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0b8070690 0x7fe0b815b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.904+0000 7fe0b77fe640 1 --2- 172.21.15.60:0/3707108901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0b8145080 0x7fe0b815dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.904+0000 7fe0affff640 1 -- 172.21.15.60:0/3707108901 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0b8070690 msgr2=0x7fe0b815b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.904+0000 7fe0affff640 1 --2- 172.21.15.60:0/3707108901 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0b8070690 0x7fe0b815b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.904+0000 7fe0affff640 1 -- 172.21.15.60:0/3707108901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0b8145080 msgr2=0x7fe0b815dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.904+0000 7fe0affff640 1 --2- 172.21.15.60:0/3707108901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0b8145080 0x7fe0b815dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:09.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.904+0000 7fe0affff640 1 -- 172.21.15.60:0/3707108901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b813ba00 con 0x7fe0b80d0390 2024-05-09T05:01:09.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.904+0000 7fe0affff640 1 --2- 172.21.15.60:0/3707108901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0b80d0390 0x7fe0b8169970 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe0a000c460 tx=0x7fe0a000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:09.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.904+0000 7fe0af7fe640 1 -- 172.21.15.60:0/3707108901 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a0016020 con 0x7fe0b80d0390 2024-05-09T05:01:09.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.904+0000 7fe0af7fe640 1 -- 172.21.15.60:0/3707108901 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0a0012870 con 0x7fe0b80d0390 2024-05-09T05:01:09.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.904+0000 7fe0af7fe640 1 -- 172.21.15.60:0/3707108901 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a0012c60 con 0x7fe0b80d0390 2024-05-09T05:01:09.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.904+0000 7fe0bd4da640 1 -- 172.21.15.60:0/3707108901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0b813b720 con 0x7fe0b80d0390 2024-05-09T05:01:09.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.904+0000 7fe0bd4da640 1 -- 172.21.15.60:0/3707108901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0b80aec10 con 0x7fe0b80d0390 2024-05-09T05:01:09.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.905+0000 7fe0bd4da640 1 -- 172.21.15.60:0/3707108901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0b80b28f0 con 0x7fe0b80d0390 2024-05-09T05:01:09.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.906+0000 7fe0af7fe640 1 -- 172.21.15.60:0/3707108901 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe0a0019020 con 0x7fe0b80d0390 2024-05-09T05:01:09.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.906+0000 7fe0af7fe640 1 --2- 172.21.15.60:0/3707108901 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe0980762d0 0x7fe098078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:09.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.907+0000 7fe0af7fe640 1 -- 172.21.15.60:0/3707108901 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7fe0a0097030 con 0x7fe0b80d0390 2024-05-09T05:01:09.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.907+0000 7fe0b6ffd640 1 --2- 172.21.15.60:0/3707108901 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe0980762d0 0x7fe098078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:09.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.907+0000 7fe0b6ffd640 1 --2- 172.21.15.60:0/3707108901 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe0980762d0 0x7fe098078790 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe0b815c1f0 tx=0x7fe0a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:09.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:09.911+0000 7fe0af7fe640 1 -- 172.21.15.60:0/3707108901 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe0b80b28f0 con 0x7fe0b80d0390 2024-05-09T05:01:10.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.184+0000 7fe0bd4da640 1 -- 172.21.15.60:0/3707108901 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fe0b8051040 con 0x7fe0b80d0390 2024-05-09T05:01:10.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.187+0000 7fe0af7fe640 1 -- 172.21.15.60:0/3707108901 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) ==== 190+0+147 (secure 0 0 0) 0x7fe0a0020070 con 0x7fe0b80d0390 2024-05-09T05:01:10.187 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2024-05-09T05:01:10.187 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAtWDxmiEa1CBAAm5ehYW6MVg0B6WUq8TxNbg== 2024-05-09T05:01:10.187 INFO:teuthology.orchestra.run.smithi060.stdout: caps mds = "allow" 2024-05-09T05:01:10.188 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "allow r" 2024-05-09T05:01:10.188 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-05-09T05:01:10.188 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2024-05-09T05:01:10.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.189+0000 7fe0bd4da640 1 -- 172.21.15.60:0/3707108901 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe0980762d0 msgr2=0x7fe098078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:10.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.189+0000 7fe0bd4da640 1 --2- 172.21.15.60:0/3707108901 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe0980762d0 0x7fe098078790 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe0b815c1f0 tx=0x7fe0a8008040 comp rx=0 tx=0).stop 2024-05-09T05:01:10.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.189+0000 7fe0bd4da640 1 -- 172.21.15.60:0/3707108901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0b80d0390 msgr2=0x7fe0b8169970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:10.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.189+0000 7fe0bd4da640 1 --2- 172.21.15.60:0/3707108901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0b80d0390 0x7fe0b8169970 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe0a000c460 tx=0x7fe0a000c920 comp rx=0 tx=0).stop 2024-05-09T05:01:10.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.189+0000 7fe0bd4da640 1 -- 172.21.15.60:0/3707108901 shutdown_connections 2024-05-09T05:01:10.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.189+0000 7fe0bd4da640 1 --2- 172.21.15.60:0/3707108901 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe0980762d0 0x7fe098078790 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:10.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.189+0000 7fe0bd4da640 1 --2- 172.21.15.60:0/3707108901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0b8145080 0x7fe0b815dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:10.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.189+0000 7fe0bd4da640 1 --2- 172.21.15.60:0/3707108901 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0b80d0390 0x7fe0b8169970 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:10.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.189+0000 7fe0bd4da640 1 --2- 172.21.15.60:0/3707108901 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0b8070690 0x7fe0b815b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:10.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.189+0000 7fe0bd4da640 1 -- 172.21.15.60:0/3707108901 >> 172.21.15.60:0/3707108901 conn(0x7fe0b80b0da0 msgr2=0x7fe0b806f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:10.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.189+0000 7fe0bd4da640 1 -- 172.21.15.60:0/3707108901 shutdown_connections 2024-05-09T05:01:10.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.189+0000 7fe0bd4da640 1 -- 172.21.15.60:0/3707108901 wait complete. 2024-05-09T05:01:10.201 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:10.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.304+0000 7f97a8922640 1 Processor -- start 2024-05-09T05:01:10.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.304+0000 7f97a8922640 1 -- start start 2024-05-09T05:01:10.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.305+0000 7f97a8922640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f97a415c3a0 0x7f97a4070150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:10.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.305+0000 7f97a8922640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f97a40ac430 0x7f97a4070690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:10.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.305+0000 7f97a8922640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97a4097de0 0x7f97a4099a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:10.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.305+0000 7f97a8922640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f97a4071b80 con 0x7f97a415c3a0 2024-05-09T05:01:10.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.305+0000 7f97a8922640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f97a4071e60 con 0x7f97a40ac430 2024-05-09T05:01:10.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.305+0000 7f97a8922640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f97a4072140 con 0x7f97a4097de0 2024-05-09T05:01:10.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.305+0000 7f97a259c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f97a415c3a0 0x7f97a4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:10.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.305+0000 7f97a259c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f97a415c3a0 0x7f97a4070150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59828/0 (socket says 172.21.15.60:59828) 2024-05-09T05:01:10.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.305+0000 7f97a259c640 1 -- 172.21.15.60:0/685916285 learned_addr learned my addr 172.21.15.60:0/685916285 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:10.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.305+0000 7f97a2d9d640 1 --2- 172.21.15.60:0/685916285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97a4097de0 0x7f97a4099a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:10.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.305+0000 7f97a1d9b640 1 --2- 172.21.15.60:0/685916285 >> v2:172.21.15.114:3300/0 conn(0x7f97a40ac430 0x7f97a4070690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:10.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.305+0000 7f97a259c640 1 -- 172.21.15.60:0/685916285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97a4097de0 msgr2=0x7f97a4099a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:10.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.305+0000 7f97a259c640 1 --2- 172.21.15.60:0/685916285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97a4097de0 0x7f97a4099a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:10.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.305+0000 7f97a259c640 1 -- 172.21.15.60:0/685916285 >> v2:172.21.15.114:3300/0 conn(0x7f97a40ac430 msgr2=0x7f97a4070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:10.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.305+0000 7f97a259c640 1 --2- 172.21.15.60:0/685916285 >> v2:172.21.15.114:3300/0 conn(0x7f97a40ac430 0x7f97a4070690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:10.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.305+0000 7f97a259c640 1 -- 172.21.15.60:0/685916285 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a4072420 con 0x7f97a415c3a0 2024-05-09T05:01:10.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.306+0000 7f97a259c640 1 --2- 172.21.15.60:0/685916285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97a415c3a0 0x7f97a4070150 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f97a406a150 tx=0x7f979402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e6429f74a6e8b57 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:10.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.306+0000 7f97a159a640 1 -- 172.21.15.60:0/685916285 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9794037070 con 0x7f97a415c3a0 2024-05-09T05:01:10.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.306+0000 7f97a159a640 1 -- 172.21.15.60:0/685916285 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9794032440 con 0x7f97a415c3a0 2024-05-09T05:01:10.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.306+0000 7f97a159a640 1 -- 172.21.15.60:0/685916285 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9794032810 con 0x7f97a415c3a0 2024-05-09T05:01:10.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.306+0000 7f97a8922640 1 -- 172.21.15.60:0/685916285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97a415c3a0 msgr2=0x7f97a4070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:10.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.306+0000 7f97a8922640 1 --2- 172.21.15.60:0/685916285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97a415c3a0 0x7f97a4070150 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f97a406a150 tx=0x7f979402a1e0 comp rx=0 tx=0).stop 2024-05-09T05:01:10.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.307+0000 7f97a8922640 1 -- 172.21.15.60:0/685916285 shutdown_connections 2024-05-09T05:01:10.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.307+0000 7f97a8922640 1 --2- 172.21.15.60:0/685916285 >> v2:172.21.15.114:3300/0 conn(0x7f97a40ac430 0x7f97a4070690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:10.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.307+0000 7f97a8922640 1 --2- 172.21.15.60:0/685916285 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97a4097de0 0x7f97a4099a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:10.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.307+0000 7f97a8922640 1 --2- 172.21.15.60:0/685916285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97a415c3a0 0x7f97a4070150 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:10.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.307+0000 7f97a8922640 1 -- 172.21.15.60:0/685916285 >> 172.21.15.60:0/685916285 conn(0x7f97a40972f0 msgr2=0x7f97a40976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:10.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.307+0000 7f97a8922640 1 -- 172.21.15.60:0/685916285 shutdown_connections 2024-05-09T05:01:10.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.307+0000 7f97a8922640 1 -- 172.21.15.60:0/685916285 wait complete. 2024-05-09T05:01:10.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.308+0000 7f97a8922640 1 Processor -- start 2024-05-09T05:01:10.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.308+0000 7f97a8922640 1 -- start start 2024-05-09T05:01:10.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.309+0000 7f97a8922640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97a4097de0 0x7f97a40b0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:10.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.309+0000 7f97a8922640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97a40ac430 0x7f97a4157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:10.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.309+0000 7f97a8922640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97a415c3a0 0x7f97a4158490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:10.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.309+0000 7f97a8922640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f97a4071810 con 0x7f97a40ac430 2024-05-09T05:01:10.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.309+0000 7f97a8922640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f97a415c780 con 0x7f97a415c3a0 2024-05-09T05:01:10.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.309+0000 7f97a8922640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f97a4099fb0 con 0x7f97a4097de0 2024-05-09T05:01:10.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.309+0000 7f97a1d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97a40ac430 0x7f97a4157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:10.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.309+0000 7f97a1d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97a40ac430 0x7f97a4157f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59830/0 (socket says 172.21.15.60:59830) 2024-05-09T05:01:10.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.309+0000 7f97a1d9b640 1 -- 172.21.15.60:0/4279104500 learned_addr learned my addr 172.21.15.60:0/4279104500 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:10.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.309+0000 7f97a259c640 1 --2- 172.21.15.60:0/4279104500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97a4097de0 0x7f97a40b0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:10.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.309+0000 7f97a1d9b640 1 -- 172.21.15.60:0/4279104500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97a4097de0 msgr2=0x7f97a40b0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:10.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.309+0000 7f97a1d9b640 1 --2- 172.21.15.60:0/4279104500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97a4097de0 0x7f97a40b0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:10.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.309+0000 7f97a2d9d640 1 --2- 172.21.15.60:0/4279104500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97a415c3a0 0x7f97a4158490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:10.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.309+0000 7f97a1d9b640 1 -- 172.21.15.60:0/4279104500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97a415c3a0 msgr2=0x7f97a4158490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:10.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.309+0000 7f97a1d9b640 1 --2- 172.21.15.60:0/4279104500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97a415c3a0 0x7f97a4158490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:10.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.309+0000 7f97a1d9b640 1 -- 172.21.15.60:0/4279104500 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a40ad7c0 con 0x7f97a40ac430 2024-05-09T05:01:10.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.309+0000 7f97a1d9b640 1 --2- 172.21.15.60:0/4279104500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97a40ac430 0x7f97a4157f50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f978c00d7d0 tx=0x7f978c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:10.314 INFO:tasks.ceph.mds.c.smithi060.stderr:2024-05-09T05:01:10.309+0000 7f1b3b1e4640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T05:01:10.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.310+0000 7f979affd640 1 -- 172.21.15.60:0/4279104500 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f978c00b660 con 0x7f97a40ac430 2024-05-09T05:01:10.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.310+0000 7f979affd640 1 -- 172.21.15.60:0/4279104500 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f978c019070 con 0x7f97a40ac430 2024-05-09T05:01:10.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.310+0000 7f97a8922640 1 -- 172.21.15.60:0/4279104500 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a40ada80 con 0x7f97a40ac430 2024-05-09T05:01:10.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.310+0000 7f979affd640 1 -- 172.21.15.60:0/4279104500 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f978c00ba60 con 0x7f97a40ac430 2024-05-09T05:01:10.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.310+0000 7f97a8922640 1 -- 172.21.15.60:0/4279104500 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a40ae040 con 0x7f97a40ac430 2024-05-09T05:01:10.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.311+0000 7f97a8922640 1 -- 172.21.15.60:0/4279104500 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a4051040 con 0x7f97a40ac430 2024-05-09T05:01:10.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.311+0000 7f979affd640 1 -- 172.21.15.60:0/4279104500 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f978c00bc00 con 0x7f97a40ac430 2024-05-09T05:01:10.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.312+0000 7f979affd640 1 --2- 172.21.15.60:0/4279104500 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f97840762d0 0x7f9784078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:10.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.312+0000 7f979affd640 1 -- 172.21.15.60:0/4279104500 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7f978c097bf0 con 0x7f97a40ac430 2024-05-09T05:01:10.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.312+0000 7f97a259c640 1 --2- 172.21.15.60:0/4279104500 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f97840762d0 0x7f9784078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:10.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.312+0000 7f97a259c640 1 --2- 172.21.15.60:0/4279104500 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f97840762d0 0x7f9784078790 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9794034000 tx=0x7f979402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:10.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.316+0000 7f979affd640 1 -- 172.21.15.60:0/4279104500 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f978c0613c0 con 0x7f97a40ac430 2024-05-09T05:01:10.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.543+0000 7f97a8922640 1 -- 172.21.15.60:0/4279104500 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97a4159340 con 0x7f97a40ac430 2024-05-09T05:01:10.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.544+0000 7f979affd640 1 -- 172.21.15.60:0/4279104500 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5181 (secure 0 0 0) 0x7f978c060cc0 con 0x7f97a40ac430 2024-05-09T05:01:10.544 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:10.545 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4547,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2372252858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2372252858},{"type":"v1","addr":"172.21.15.114:6836","nonce":2372252858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1761282610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1761282610},{"type":"v1","addr":"172.21.15.114:6837","nonce":1761282610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4765,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/300408012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":300408012},{"type":"v1","addr":"172.21.15.60:6835","nonce":300408012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:04.698975+0000","modified":"2024-05-09T05:01:07.722167+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4893},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4893":{"gid":4893,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/399160914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":399160914},{"type":"v1","addr":"172.21.15.60:6834","nonce":399160914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4893,"qdb_cluster":[4893]},"id":2}]} 2024-05-09T05:01:10.545 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 17 2024-05-09T05:01:10.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.546+0000 7f97a8922640 1 -- 172.21.15.60:0/4279104500 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f97840762d0 msgr2=0x7f9784078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:10.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.546+0000 7f97a8922640 1 --2- 172.21.15.60:0/4279104500 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f97840762d0 0x7f9784078790 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9794034000 tx=0x7f979402e000 comp rx=0 tx=0).stop 2024-05-09T05:01:10.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.546+0000 7f97a8922640 1 -- 172.21.15.60:0/4279104500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97a40ac430 msgr2=0x7f97a4157f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:10.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.546+0000 7f97a8922640 1 --2- 172.21.15.60:0/4279104500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97a40ac430 0x7f97a4157f50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f978c00d7d0 tx=0x7f978c00dc90 comp rx=0 tx=0).stop 2024-05-09T05:01:10.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.546+0000 7f97a8922640 1 -- 172.21.15.60:0/4279104500 shutdown_connections 2024-05-09T05:01:10.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.546+0000 7f97a8922640 1 --2- 172.21.15.60:0/4279104500 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f97840762d0 0x7f9784078790 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:10.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.546+0000 7f97a8922640 1 --2- 172.21.15.60:0/4279104500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97a415c3a0 0x7f97a4158490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:10.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.546+0000 7f97a8922640 1 --2- 172.21.15.60:0/4279104500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97a40ac430 0x7f97a4157f50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:10.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.546+0000 7f97a8922640 1 --2- 172.21.15.60:0/4279104500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f97a4097de0 0x7f97a40b0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:10.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.546+0000 7f97a8922640 1 -- 172.21.15.60:0/4279104500 >> 172.21.15.60:0/4279104500 conn(0x7f97a40972f0 msgr2=0x7f97a40976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:10.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.546+0000 7f97a8922640 1 -- 172.21.15.60:0/4279104500 shutdown_connections 2024-05-09T05:01:10.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:10.546+0000 7f97a8922640 1 -- 172.21.15.60:0/4279104500 wait complete. 2024-05-09T05:01:10.961 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T05:01:04.698975+0000', 'modified': '2024-05-09T05:01:07.722167+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4893}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4893': {'gid': 4893, 'name': 'c', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6834/399160914', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 399160914}, {'type': 'v1', 'addr': '172.21.15.60:6834', 'nonce': 399160914}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4893, 'qdb_cluster': [4893]} 2024-05-09T05:01:10.961 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T05:01:10.961 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T05:01:11.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.071+0000 7f40f6045640 1 Processor -- start 2024-05-09T05:01:11.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.071+0000 7f40f6045640 1 -- start start 2024-05-09T05:01:11.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.071+0000 7f40f6045640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40f015f300 0x7f40f01639f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.071+0000 7f40f6045640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f40f006f9f0 0x7f40f00d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.071+0000 7f40f6045640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40f006ff60 con 0x7f40f00d3930 2024-05-09T05:01:11.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.072+0000 7f40f6045640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f40f0070240 con 0x7f40f006f9f0 2024-05-09T05:01:11.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.072+0000 7f40f6045640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f40f0070520 con 0x7f40f015f300 2024-05-09T05:01:11.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.072+0000 7f40eeffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f40f006f9f0 0x7f40f00d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.072+0000 7f40eeffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f40f006f9f0 0x7f40f00d33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59846/0 (socket says 172.21.15.60:59846) 2024-05-09T05:01:11.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.072+0000 7f40eeffd640 1 -- 172.21.15.60:0/3054727618 learned_addr learned my addr 172.21.15.60:0/3054727618 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:11.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.072+0000 7f40ef7fe640 1 --2- 172.21.15.60:0/3054727618 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40f015f300 0x7f40f01639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.072+0000 7f40eeffd640 1 -- 172.21.15.60:0/3054727618 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40f015f300 msgr2=0x7f40f01639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:11.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.072+0000 7f40eeffd640 1 --2- 172.21.15.60:0/3054727618 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40f015f300 0x7f40f01639f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.072+0000 7f40eeffd640 1 -- 172.21.15.60:0/3054727618 >> v1:172.21.15.114:6789/0 conn(0x7f40f00d3930 legacy=0x7f40f006ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:11.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.072+0000 7f40eeffd640 1 -- 172.21.15.60:0/3054727618 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40f006e740 con 0x7f40f006f9f0 2024-05-09T05:01:11.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.073+0000 7f40eeffd640 1 --2- 172.21.15.60:0/3054727618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40f006f9f0 0x7f40f00d33f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f40e4009870 tx=0x7f40e402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34011c1a99b467cd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:11.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.073+0000 7f40ee7fc640 1 -- 172.21.15.60:0/3054727618 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e4037070 con 0x7f40f006f9f0 2024-05-09T05:01:11.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.073+0000 7f40ee7fc640 1 -- 172.21.15.60:0/3054727618 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40e4032440 con 0x7f40f006f9f0 2024-05-09T05:01:11.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.073+0000 7f40ee7fc640 1 -- 172.21.15.60:0/3054727618 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e4032810 con 0x7f40f006f9f0 2024-05-09T05:01:11.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.074+0000 7f40f6045640 1 -- 172.21.15.60:0/3054727618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40f006f9f0 msgr2=0x7f40f00d33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:11.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.074+0000 7f40f6045640 1 --2- 172.21.15.60:0/3054727618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40f006f9f0 0x7f40f00d33f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f40e4009870 tx=0x7f40e402a270 comp rx=0 tx=0).stop 2024-05-09T05:01:11.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.074+0000 7f40f6045640 1 -- 172.21.15.60:0/3054727618 shutdown_connections 2024-05-09T05:01:11.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.074+0000 7f40f6045640 1 --2- 172.21.15.60:0/3054727618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40f006f9f0 0x7f40f00d33f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.074+0000 7f40f6045640 1 --2- 172.21.15.60:0/3054727618 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40f015f300 0x7f40f01639f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.074+0000 7f40f6045640 1 -- 172.21.15.60:0/3054727618 >> 172.21.15.60:0/3054727618 conn(0x7f40f009b850 msgr2=0x7f40f009bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:11.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.074+0000 7f40f6045640 1 -- 172.21.15.60:0/3054727618 shutdown_connections 2024-05-09T05:01:11.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.074+0000 7f40f6045640 1 -- 172.21.15.60:0/3054727618 wait complete. 2024-05-09T05:01:11.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.075+0000 7f40f6045640 1 Processor -- start 2024-05-09T05:01:11.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.075+0000 7f40f6045640 1 -- start start 2024-05-09T05:01:11.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40f6045640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40f006f9f0 0x7f40f00ae830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40f6045640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40f00d3930 0x7f40f015a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40f6045640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40f015f300 0x7f40f00afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40f6045640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f40f006f650 con 0x7f40f00d3930 2024-05-09T05:01:11.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40f6045640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f40f0163f30 con 0x7f40f006f9f0 2024-05-09T05:01:11.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40f6045640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f40f006f7d0 con 0x7f40f015f300 2024-05-09T05:01:11.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40eeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40f00d3930 0x7f40f015a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40eeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40f00d3930 0x7f40f015a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59852/0 (socket says 172.21.15.60:59852) 2024-05-09T05:01:11.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40eeffd640 1 -- 172.21.15.60:0/3613949871 learned_addr learned my addr 172.21.15.60:0/3613949871 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:11.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40ef7fe640 1 --2- 172.21.15.60:0/3613949871 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40f006f9f0 0x7f40f00ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40effff640 1 --2- 172.21.15.60:0/3613949871 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40f015f300 0x7f40f00afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40eeffd640 1 -- 172.21.15.60:0/3613949871 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40f015f300 msgr2=0x7f40f00afec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:11.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40eeffd640 1 --2- 172.21.15.60:0/3613949871 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40f015f300 0x7f40f00afec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40eeffd640 1 -- 172.21.15.60:0/3613949871 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40f006f9f0 msgr2=0x7f40f00ae830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:11.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40eeffd640 1 --2- 172.21.15.60:0/3613949871 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40f006f9f0 0x7f40f00ae830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40eeffd640 1 -- 172.21.15.60:0/3613949871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40f00ad790 con 0x7f40f00d3930 2024-05-09T05:01:11.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40effff640 1 --2- 172.21.15.60:0/3613949871 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40f015f300 0x7f40f00afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:11.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40eeffd640 1 --2- 172.21.15.60:0/3613949871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40f00d3930 0x7f40f015a1b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f40e4000c00 tx=0x7f40e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:11.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40d7fff640 1 -- 172.21.15.60:0/3613949871 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e4037040 con 0x7f40f00d3930 2024-05-09T05:01:11.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40d7fff640 1 -- 172.21.15.60:0/3613949871 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40e4042070 con 0x7f40f00d3930 2024-05-09T05:01:11.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.076+0000 7f40f6045640 1 -- 172.21.15.60:0/3613949871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40f00ada70 con 0x7f40f00d3930 2024-05-09T05:01:11.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.077+0000 7f40d7fff640 1 -- 172.21.15.60:0/3613949871 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e4030880 con 0x7f40f00d3930 2024-05-09T05:01:11.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.077+0000 7f40f6045640 1 -- 172.21.15.60:0/3613949871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40f00ae030 con 0x7f40f00d3930 2024-05-09T05:01:11.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.079+0000 7f40d7fff640 1 -- 172.21.15.60:0/3613949871 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f40e4030a20 con 0x7f40f00d3930 2024-05-09T05:01:11.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.079+0000 7f40d7fff640 1 --2- 172.21.15.60:0/3613949871 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f40dc0762d0 0x7f40dc078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.079+0000 7f40d7fff640 1 -- 172.21.15.60:0/3613949871 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7f40e40b7fe0 con 0x7f40f00d3930 2024-05-09T05:01:11.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.080+0000 7f40ef7fe640 1 --2- 172.21.15.60:0/3613949871 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f40dc0762d0 0x7f40dc078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.080+0000 7f40f89fd740 1 Processor -- start 2024-05-09T05:01:11.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.080+0000 7f40f89fd740 1 -- start start 2024-05-09T05:01:11.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.080+0000 7f40ef7fe640 1 --2- 172.21.15.60:0/3613949871 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f40dc0762d0 0x7f40dc078790 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f40e0009a30 tx=0x7f40e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:11.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40f89fd740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee43320ff0 0x55ee43323800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40f89fd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee433213d0 0x55ee43323d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40f89fd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee432f3c50 0x55ee432f7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40f89fd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ee4325d8d0 con 0x55ee433213d0 2024-05-09T05:01:11.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40f89fd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ee43216c10 con 0x55ee432f3c50 2024-05-09T05:01:11.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40f89fd740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x55ee43259890 con 0x55ee43320ff0 2024-05-09T05:01:11.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40eeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee433213d0 0x55ee43323d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40eeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee433213d0 0x55ee43323d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59868/0 (socket says 172.21.15.60:59868) 2024-05-09T05:01:11.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40eeffd640 1 -- 172.21.15.60:0/688071154 learned_addr learned my addr 172.21.15.60:0/688071154 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:11.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40ef7fe640 1 --2- 172.21.15.60:0/688071154 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee43320ff0 0x55ee43323800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40effff640 1 --2- 172.21.15.60:0/688071154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee432f3c50 0x55ee432f7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40eeffd640 1 -- 172.21.15.60:0/688071154 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee43320ff0 msgr2=0x55ee43323800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:11.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40eeffd640 1 --2- 172.21.15.60:0/688071154 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee43320ff0 0x55ee43323800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40eeffd640 1 -- 172.21.15.60:0/688071154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee432f3c50 msgr2=0x55ee432f7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:11.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40eeffd640 1 --2- 172.21.15.60:0/688071154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee432f3c50 0x55ee432f7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40eeffd640 1 -- 172.21.15.60:0/688071154 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee4321e5d0 con 0x55ee433213d0 2024-05-09T05:01:11.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40eeffd640 1 --2- 172.21.15.60:0/688071154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee433213d0 0x55ee43323d40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f40e4035040 tx=0x7f40e4080a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b279f41fff5295c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40d57fa640 1 -- 172.21.15.60:0/688071154 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e4080bc0 con 0x55ee433213d0 2024-05-09T05:01:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40d57fa640 1 -- 172.21.15.60:0/688071154 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40e40817b0 con 0x55ee433213d0 2024-05-09T05:01:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.081+0000 7f40d57fa640 1 -- 172.21.15.60:0/688071154 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e40810b0 con 0x55ee433213d0 2024-05-09T05:01:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.082+0000 7f40f89fd740 1 -- 172.21.15.60:0/688071154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee433213d0 msgr2=0x55ee43323d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.082+0000 7f40f89fd740 1 --2- 172.21.15.60:0/688071154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee433213d0 0x55ee43323d40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f40e4035040 tx=0x7f40e4080a00 comp rx=0 tx=0).stop 2024-05-09T05:01:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.082+0000 7f40f89fd740 1 -- 172.21.15.60:0/688071154 shutdown_connections 2024-05-09T05:01:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.082+0000 7f40f89fd740 1 --2- 172.21.15.60:0/688071154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee432f3c50 0x55ee432f7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.082+0000 7f40f89fd740 1 --2- 172.21.15.60:0/688071154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee433213d0 0x55ee43323d40 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.082+0000 7f40f89fd740 1 --2- 172.21.15.60:0/688071154 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee43320ff0 0x55ee43323800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.082+0000 7f40f89fd740 1 -- 172.21.15.60:0/688071154 >> 172.21.15.60:0/688071154 conn(0x55ee432f5ec0 msgr2=0x55ee4333e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.082+0000 7f40f89fd740 1 -- 172.21.15.60:0/688071154 shutdown_connections 2024-05-09T05:01:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.082+0000 7f40f89fd740 1 -- 172.21.15.60:0/688071154 wait complete. 2024-05-09T05:01:11.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.083+0000 7f40f89fd740 1 -- start start 2024-05-09T05:01:11.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.083+0000 7f40f89fd740 1 Processor -- start 2024-05-09T05:01:11.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40f89fd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee43320ff0 0x55ee43324d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40f89fd740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee433213d0 0x55ee4331f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40f89fd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee43320060 0x55ee433468a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40f89fd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ee4325d8d0 con 0x55ee43320060 2024-05-09T05:01:11.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40f89fd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ee4321a760 con 0x55ee43320ff0 2024-05-09T05:01:11.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40f89fd740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x55ee432f9070 con 0x55ee433213d0 2024-05-09T05:01:11.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40f89fd740 10 client.0 _ll_register_callbacks cb 0x55ee43253cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:01:11.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40effff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee43320060 0x55ee433468a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40effff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee43320060 0x55ee433468a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59870/0 (socket says 172.21.15.60:59870) 2024-05-09T05:01:11.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40effff640 1 -- 172.21.15.60:0/3649905690 learned_addr learned my addr 172.21.15.60:0/3649905690 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:11.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40ef7fe640 1 --2- 172.21.15.60:0/3649905690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee43320ff0 0x55ee43324d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40eeffd640 1 --2- 172.21.15.60:0/3649905690 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee433213d0 0x55ee4331f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40effff640 1 -- 172.21.15.60:0/3649905690 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee433213d0 msgr2=0x55ee4331f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:11.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40effff640 1 --2- 172.21.15.60:0/3649905690 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee433213d0 0x55ee4331f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40effff640 1 -- 172.21.15.60:0/3649905690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee43320ff0 msgr2=0x55ee43324d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:11.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40effff640 1 --2- 172.21.15.60:0/3649905690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee43320ff0 0x55ee43324d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40effff640 1 -- 172.21.15.60:0/3649905690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee42f6f2d0 con 0x55ee43320060 2024-05-09T05:01:11.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40ef7fe640 1 --2- 172.21.15.60:0/3649905690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee43320ff0 0x55ee43324d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:11.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40effff640 1 --2- 172.21.15.60:0/3649905690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee43320060 0x55ee433468a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f40e8010810 tx=0x7f40e8010cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b52d86532ae81406 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:11.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40ce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-05-09T05:01:11.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40ce7fc640 1 -- 172.21.15.60:0/3649905690 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e8003b70 con 0x55ee43320060 2024-05-09T05:01:11.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40ce7fc640 1 -- 172.21.15.60:0/3649905690 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7f40e8003d10 con 0x55ee43320060 2024-05-09T05:01:11.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.084+0000 7f40f89fd740 1 -- 172.21.15.60:0/3649905690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40e40817b0 con 0x55ee43320060 2024-05-09T05:01:11.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.085+0000 7f40ce7fc640 1 -- 172.21.15.60:0/3649905690 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f40e8021050 con 0x55ee43320060 2024-05-09T05:01:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.085+0000 7f40f89fd740 10 client.5040 fetch_fsmap learned FSMap version 17 2024-05-09T05:01:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.085+0000 7f40f89fd740 1 -- 172.21.15.60:0/3649905690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55ee430396b0 con 0x55ee43320060 2024-05-09T05:01:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.085+0000 7f40ce7fc640 1 -- 172.21.15.60:0/3649905690 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40e40817b0 con 0x55ee43320060 2024-05-09T05:01:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.085+0000 7f40ce7fc640 1 -- 172.21.15.60:0/3649905690 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f40e8024020 con 0x55ee43320060 2024-05-09T05:01:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.085+0000 7f40f89fd740 10 client.5040 fetch_fsmap finished waiting for FSMap version 17 2024-05-09T05:01:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.085+0000 7f40f89fd740 10 client.5040 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T05:01:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.085+0000 7f40f89fd740 20 client.5040 populate_metadata read hostname 'smithi060' 2024-05-09T05:01:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.085+0000 7f40f89fd740 1 --2- 172.21.15.60:0/3649905690 >> [v2:172.21.15.60:6832/399160914,v1:172.21.15.60:6834/399160914] conn(0x55ee43343b20 0x55ee43343f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.085+0000 7f40f89fd740 4 client.5040 mds_command: new command op to 4893 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T05:01:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.085+0000 7f40f89fd740 1 -- 172.21.15.60:0/3649905690 --> [v2:172.21.15.60:6832/399160914,v1:172.21.15.60:6834/399160914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee431e1760 con 0x55ee43343b20 2024-05-09T05:01:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.085+0000 7f40ef7fe640 1 --2- 172.21.15.60:0/3649905690 >> [v2:172.21.15.60:6832/399160914,v1:172.21.15.60:6834/399160914] conn(0x55ee43343b20 0x55ee43343f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.085+0000 7f40ce7fc640 10 client.5040 ms_handle_connect on v2:172.21.15.60:6832/399160914 2024-05-09T05:01:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.085+0000 7f40ef7fe640 1 --2- 172.21.15.60:0/3649905690 >> [v2:172.21.15.60:6832/399160914,v1:172.21.15.60:6834/399160914] conn(0x55ee43343b20 0x55ee43343f00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9df1930c465e23 server_cookie=551029195bb54bb9 in_seq=0 out_seq=0 2024-05-09T05:01:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.086+0000 7f40ce7fc640 1 -- 172.21.15.60:0/3649905690 <== mds.0 v2:172.21.15.60:6832/399160914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ee431e1760 con 0x55ee43343b20 2024-05-09T05:01:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.086+0000 7f40ce7fc640 10 client.5040 handle_command_reply: tid=0 2024-05-09T05:01:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.086+0000 7f40f89fd740 1 client.5040 shutdown 2024-05-09T05:01:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.086+0000 7f40f89fd740 1 -- 172.21.15.60:0/3649905690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee43320060 msgr2=0x55ee433468a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.086+0000 7f40f89fd740 1 --2- 172.21.15.60:0/3649905690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee43320060 0x55ee433468a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f40e8010810 tx=0x7f40e8010cd0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.086+0000 7f40f89fd740 1 -- 172.21.15.60:0/3649905690 shutdown_connections 2024-05-09T05:01:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.086+0000 7f40f89fd740 1 --2- 172.21.15.60:0/3649905690 >> [v2:172.21.15.60:6832/399160914,v1:172.21.15.60:6834/399160914] conn(0x55ee43343b20 0x55ee43343f00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.086+0000 7f40f89fd740 1 --2- 172.21.15.60:0/3649905690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee43320060 0x55ee433468a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.086+0000 7f40f89fd740 1 --2- 172.21.15.60:0/3649905690 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee433213d0 0x55ee4331f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.086+0000 7f40f89fd740 1 --2- 172.21.15.60:0/3649905690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee43320ff0 0x55ee43324d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.086+0000 7f40f89fd740 1 -- 172.21.15.60:0/3649905690 >> 172.21.15.60:0/3649905690 conn(0x55ee432f3c50 msgr2=0x55ee433237c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:11.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.086+0000 7f40ce7fc640 0 client.5040 ms_handle_reset on v2:172.21.15.60:6832/399160914 2024-05-09T05:01:11.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.086+0000 7f40f89fd740 1 -- 172.21.15.60:0/3649905690 shutdown_connections 2024-05-09T05:01:11.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.086+0000 7f40f89fd740 1 -- 172.21.15.60:0/3649905690 wait complete. 2024-05-09T05:01:11.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.086+0000 7f40f89fd740 20 client.5040 trim_cache size 0 max 16384 2024-05-09T05:01:11.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.105+0000 7f40f89fd740 1 Processor -- start 2024-05-09T05:01:11.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.105+0000 7f40f89fd740 1 -- start start 2024-05-09T05:01:11.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.106+0000 7f40f89fd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee43320060 0x55ee433196c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.106+0000 7f40f89fd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee43320ff0 0x55ee432f43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.106+0000 7f40f89fd740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee433213d0 0x55ee432f4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.106+0000 7f40f89fd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f40f0201050 con 0x55ee43320060 2024-05-09T05:01:11.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.106+0000 7f40f89fd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f40f00b10a0 con 0x55ee43320ff0 2024-05-09T05:01:11.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.106+0000 7f40f89fd740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f40f00b16f0 con 0x55ee433213d0 2024-05-09T05:01:11.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.106+0000 7f40ef7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee43320060 0x55ee433196c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.106+0000 7f40ef7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee43320060 0x55ee433196c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59874/0 (socket says 172.21.15.60:59874) 2024-05-09T05:01:11.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.106+0000 7f40ef7fe640 1 -- 172.21.15.60:0/3451261999 learned_addr learned my addr 172.21.15.60:0/3451261999 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:11.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.106+0000 7f40ef7fe640 1 -- 172.21.15.60:0/3451261999 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee433213d0 msgr2=0x55ee432f4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:11.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.106+0000 7f40ef7fe640 1 --2- 172.21.15.60:0/3451261999 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee433213d0 0x55ee432f4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.106+0000 7f40ef7fe640 1 -- 172.21.15.60:0/3451261999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee43320ff0 msgr2=0x55ee432f43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:11.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.106+0000 7f40ef7fe640 1 --2- 172.21.15.60:0/3451261999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee43320ff0 0x55ee432f43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.106+0000 7f40ef7fe640 1 -- 172.21.15.60:0/3451261999 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee432f41d0 con 0x55ee43320060 2024-05-09T05:01:11.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.106+0000 7f40ef7fe640 1 --2- 172.21.15.60:0/3451261999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee43320060 0x55ee433196c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f40e0011d30 tx=0x7f40e0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be11a5d8f8facbe1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:11.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.107+0000 7f40d4ff9640 1 -- 172.21.15.60:0/3451261999 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e0031070 con 0x55ee43320060 2024-05-09T05:01:11.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.107+0000 7f40d4ff9640 1 -- 172.21.15.60:0/3451261999 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40e001a040 con 0x55ee43320060 2024-05-09T05:01:11.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.107+0000 7f40d4ff9640 1 -- 172.21.15.60:0/3451261999 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e0004230 con 0x55ee43320060 2024-05-09T05:01:11.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.107+0000 7f40f89fd740 1 -- 172.21.15.60:0/3451261999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee43320060 msgr2=0x55ee433196c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:11.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.107+0000 7f40f89fd740 1 --2- 172.21.15.60:0/3451261999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee43320060 0x55ee433196c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f40e0011d30 tx=0x7f40e0014000 comp rx=0 tx=0).stop 2024-05-09T05:01:11.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.107+0000 7f40f89fd740 1 -- 172.21.15.60:0/3451261999 shutdown_connections 2024-05-09T05:01:11.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.107+0000 7f40f89fd740 1 --2- 172.21.15.60:0/3451261999 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee433213d0 0x55ee432f4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.107+0000 7f40f89fd740 1 --2- 172.21.15.60:0/3451261999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee43320ff0 0x55ee432f43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.107+0000 7f40f89fd740 1 --2- 172.21.15.60:0/3451261999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee43320060 0x55ee433196c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.107+0000 7f40f89fd740 1 -- 172.21.15.60:0/3451261999 >> 172.21.15.60:0/3451261999 conn(0x55ee432f3c50 msgr2=0x55ee43346880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:11.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.107+0000 7f40f89fd740 1 -- 172.21.15.60:0/3451261999 shutdown_connections 2024-05-09T05:01:11.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.107+0000 7f40f89fd740 1 -- 172.21.15.60:0/3451261999 wait complete. 2024-05-09T05:01:11.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.108+0000 7f40f89fd740 1 -- start start 2024-05-09T05:01:11.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.108+0000 7f40f89fd740 1 Processor -- start 2024-05-09T05:01:11.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40f89fd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee43320060 0x55ee4333d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40f89fd740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee43320ff0 0x55ee4331e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40f89fd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee433213d0 0x55ee433462c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40f89fd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55ee43326670 con 0x55ee433213d0 2024-05-09T05:01:11.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40f89fd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ee43345dc0 con 0x55ee43320060 2024-05-09T05:01:11.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40f89fd740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x55ee43341fc0 con 0x55ee43320ff0 2024-05-09T05:01:11.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40f89fd740 10 client.0 _ll_register_callbacks cb 0x7f40e8001a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:01:11.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40effff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee433213d0 0x55ee433462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40effff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee433213d0 0x55ee433462c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59880/0 (socket says 172.21.15.60:59880) 2024-05-09T05:01:11.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40effff640 1 -- 172.21.15.60:0/3006025301 learned_addr learned my addr 172.21.15.60:0/3006025301 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:11.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40eeffd640 1 --2- 172.21.15.60:0/3006025301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee43320ff0 0x55ee4331e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40ef7fe640 1 --2- 172.21.15.60:0/3006025301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee43320060 0x55ee4333d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40effff640 1 -- 172.21.15.60:0/3006025301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee43320ff0 msgr2=0x55ee4331e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:11.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40effff640 1 --2- 172.21.15.60:0/3006025301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee43320ff0 0x55ee4331e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40effff640 1 -- 172.21.15.60:0/3006025301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee43320060 msgr2=0x55ee4333d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:11.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40effff640 1 --2- 172.21.15.60:0/3006025301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee43320060 0x55ee4333d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40effff640 1 -- 172.21.15.60:0/3006025301 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee4331d900 con 0x55ee433213d0 2024-05-09T05:01:11.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40eeffd640 1 --2- 172.21.15.60:0/3006025301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee43320ff0 0x55ee4331e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:11.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40effff640 1 --2- 172.21.15.60:0/3006025301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee433213d0 0x55ee433462c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f40e8003c50 tx=0x7f40e8001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db62128899f5a0c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:11.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40ce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-05-09T05:01:11.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40ce7fc640 1 -- 172.21.15.60:0/3006025301 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e8029070 con 0x55ee433213d0 2024-05-09T05:01:11.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40ce7fc640 1 -- 172.21.15.60:0/3006025301 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7f40e800e9a0 con 0x55ee433213d0 2024-05-09T05:01:11.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.109+0000 7f40f89fd740 1 -- 172.21.15.60:0/3006025301 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40e001a040 con 0x55ee433213d0 2024-05-09T05:01:11.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.110+0000 7f40ce7fc640 1 -- 172.21.15.60:0/3006025301 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f40e8036050 con 0x55ee433213d0 2024-05-09T05:01:11.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.110+0000 7f40f89fd740 10 client.5046 fetch_fsmap learned FSMap version 17 2024-05-09T05:01:11.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.110+0000 7f40f89fd740 1 -- 172.21.15.60:0/3006025301 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55ee432f2ae0 con 0x55ee433213d0 2024-05-09T05:01:11.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.110+0000 7f40ce7fc640 1 -- 172.21.15.60:0/3006025301 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40e001a040 con 0x55ee433213d0 2024-05-09T05:01:11.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.110+0000 7f40ce7fc640 1 -- 172.21.15.60:0/3006025301 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f40e8039020 con 0x55ee433213d0 2024-05-09T05:01:11.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.110+0000 7f40f89fd740 10 client.5046 fetch_fsmap finished waiting for FSMap version 17 2024-05-09T05:01:11.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.110+0000 7f40f89fd740 10 client.5046 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T05:01:11.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.110+0000 7f40f89fd740 20 client.5046 populate_metadata read hostname 'smithi060' 2024-05-09T05:01:11.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.110+0000 7f40f89fd740 1 --2- 172.21.15.60:0/3006025301 >> [v2:172.21.15.60:6832/399160914,v1:172.21.15.60:6834/399160914] conn(0x55ee433617c0 0x55ee43361ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.110+0000 7f40f89fd740 4 client.5046 mds_command: new command op to 4893 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T05:01:11.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.110+0000 7f40f89fd740 1 -- 172.21.15.60:0/3006025301 --> [v2:172.21.15.60:6832/399160914,v1:172.21.15.60:6834/399160914] -- command(tid 0: {"prefix": "status"}) -- 0x55ee4326b4e0 con 0x55ee433617c0 2024-05-09T05:01:11.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.110+0000 7f40ef7fe640 1 --2- 172.21.15.60:0/3006025301 >> [v2:172.21.15.60:6832/399160914,v1:172.21.15.60:6834/399160914] conn(0x55ee433617c0 0x55ee43361ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.110+0000 7f40ce7fc640 10 client.5046 ms_handle_connect on v2:172.21.15.60:6832/399160914 2024-05-09T05:01:11.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.110+0000 7f40ef7fe640 1 --2- 172.21.15.60:0/3006025301 >> [v2:172.21.15.60:6832/399160914,v1:172.21.15.60:6834/399160914] conn(0x55ee433617c0 0x55ee43361ba0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb1768908f49fef6 server_cookie=a94001182d1253bc in_seq=0 out_seq=0 2024-05-09T05:01:11.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40ce7fc640 1 -- 172.21.15.60:0/3006025301 <== mds.0 v2:172.21.15.60:6832/399160914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55ee4326b4e0 con 0x55ee433617c0 2024-05-09T05:01:11.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40ce7fc640 10 client.5046 handle_command_reply: tid=0 2024-05-09T05:01:11.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f89fd740 1 client.5046 shutdown 2024-05-09T05:01:11.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f89fd740 1 -- 172.21.15.60:0/3006025301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee433213d0 msgr2=0x55ee433462c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:11.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f89fd740 1 --2- 172.21.15.60:0/3006025301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee433213d0 0x55ee433462c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f40e8003c50 tx=0x7f40e8001250 comp rx=0 tx=0).stop 2024-05-09T05:01:11.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f89fd740 1 -- 172.21.15.60:0/3006025301 shutdown_connections 2024-05-09T05:01:11.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f89fd740 1 --2- 172.21.15.60:0/3006025301 >> [v2:172.21.15.60:6832/399160914,v1:172.21.15.60:6834/399160914] conn(0x55ee433617c0 0x55ee43361ba0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f89fd740 1 --2- 172.21.15.60:0/3006025301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ee433213d0 0x55ee433462c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40ce7fc640 0 client.5046 ms_handle_reset on v2:172.21.15.60:6832/399160914 2024-05-09T05:01:11.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f89fd740 1 --2- 172.21.15.60:0/3006025301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55ee43320ff0 0x55ee4331e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f89fd740 1 --2- 172.21.15.60:0/3006025301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ee43320060 0x55ee4333d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f89fd740 1 -- 172.21.15.60:0/3006025301 >> 172.21.15.60:0/3006025301 conn(0x55ee432f3c50 msgr2=0x55ee432f43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:11.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f89fd740 1 -- 172.21.15.60:0/3006025301 shutdown_connections 2024-05-09T05:01:11.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f89fd740 1 -- 172.21.15.60:0/3006025301 wait complete. 2024-05-09T05:01:11.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f89fd740 20 client.5046 trim_cache size 0 max 16384 2024-05-09T05:01:11.120 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:01:11.120 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2", 2024-05-09T05:01:11.120 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2024-05-09T05:01:11.120 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4893, 2024-05-09T05:01:11.120 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2024-05-09T05:01:11.120 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2024-05-09T05:01:11.121 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2024-05-09T05:01:11.121 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 5.7727509169999998, 2024-05-09T05:01:11.121 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 17, 2024-05-09T05:01:11.121 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 36, 2024-05-09T05:01:11.121 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 36, 2024-05-09T05:01:11.121 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 12.657451996000001 2024-05-09T05:01:11.121 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:01:11.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f6045640 1 -- 172.21.15.60:0/3613949871 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f40dc0762d0 msgr2=0x7f40dc078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:11.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f6045640 1 --2- 172.21.15.60:0/3613949871 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f40dc0762d0 0x7f40dc078790 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f40e0009a30 tx=0x7f40e0008040 comp rx=0 tx=0).stop 2024-05-09T05:01:11.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f6045640 1 -- 172.21.15.60:0/3613949871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40f00d3930 msgr2=0x7f40f015a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:11.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f6045640 1 --2- 172.21.15.60:0/3613949871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40f00d3930 0x7f40f015a1b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f40e4000c00 tx=0x7f40e40046c0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f6045640 1 -- 172.21.15.60:0/3613949871 shutdown_connections 2024-05-09T05:01:11.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f6045640 1 --2- 172.21.15.60:0/3613949871 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f40dc0762d0 0x7f40dc078790 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f6045640 1 --2- 172.21.15.60:0/3613949871 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f40f015f300 0x7f40f00afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f6045640 1 --2- 172.21.15.60:0/3613949871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40f00d3930 0x7f40f015a1b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f6045640 1 --2- 172.21.15.60:0/3613949871 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40f006f9f0 0x7f40f00ae830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.111+0000 7f40f6045640 1 -- 172.21.15.60:0/3613949871 >> 172.21.15.60:0/3613949871 conn(0x7f40f009b850 msgr2=0x7f40f00d2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:11.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.112+0000 7f40f6045640 1 -- 172.21.15.60:0/3613949871 shutdown_connections 2024-05-09T05:01:11.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.112+0000 7f40f6045640 1 -- 172.21.15.60:0/3613949871 wait complete. 2024-05-09T05:01:11.124 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T05:01:11.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.227+0000 7f48d1b19640 1 Processor -- start 2024-05-09T05:01:11.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.227+0000 7f48d1b19640 1 -- start start 2024-05-09T05:01:11.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.227+0000 7f48d1b19640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f48cc15f3e0 0x7f48cc0b4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.228+0000 7f48d1b19640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48cc06f590 con 0x7f48cc073310 2024-05-09T05:01:11.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.228+0000 7f48d1b19640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48cc071f30 con 0x7f48cc09c8b0 2024-05-09T05:01:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.228+0000 7f48d1b19640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f48cc071ce0 con 0x7f48cc15f3e0 2024-05-09T05:01:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.228+0000 7f48cb7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f48cc15f3e0 0x7f48cc0b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.228+0000 7f48cbfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f48cc073310 0x7f48cc06ee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33546/0 (socket says 172.21.15.60:33546) 2024-05-09T05:01:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.228+0000 7f48cb7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f48cc15f3e0 0x7f48cc0b4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59888/0 (socket says 172.21.15.60:59888) 2024-05-09T05:01:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.228+0000 7f48cbfff640 1 -- 172.21.15.60:0/4997976 learned_addr learned my addr 172.21.15.60:0/4997976 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.228+0000 7f48cb7fe640 1 -- 172.21.15.60:0/4997976 >> v1:172.21.15.114:6789/0 conn(0x7f48cc09c8b0 legacy=0x7f48cc06e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.228+0000 7f48cb7fe640 1 -- 172.21.15.60:0/4997976 >> v1:172.21.15.60:6789/0 conn(0x7f48cc073310 legacy=0x7f48cc06ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.228+0000 7f48cb7fe640 1 -- 172.21.15.60:0/4997976 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48cc072210 con 0x7f48cc15f3e0 2024-05-09T05:01:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.229+0000 7f48cb7fe640 1 --2- 172.21.15.60:0/4997976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48cc15f3e0 0x7f48cc0b4630 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f48bc0097a0 tx=0x7f48bc02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb7bc09285536f59 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.229+0000 7f48ca7fc640 1 -- 172.21.15.60:0/4997976 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48bc037070 con 0x7f48cc15f3e0 2024-05-09T05:01:11.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.229+0000 7f48ca7fc640 1 -- 172.21.15.60:0/4997976 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48bc032440 con 0x7f48cc15f3e0 2024-05-09T05:01:11.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.229+0000 7f48ca7fc640 1 -- 172.21.15.60:0/4997976 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48bc032810 con 0x7f48cc15f3e0 2024-05-09T05:01:11.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.229+0000 7f48d1b19640 1 -- 172.21.15.60:0/4997976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48cc15f3e0 msgr2=0x7f48cc0b4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.229+0000 7f48d1b19640 1 --2- 172.21.15.60:0/4997976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48cc15f3e0 0x7f48cc0b4630 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f48bc0097a0 tx=0x7f48bc02a180 comp rx=0 tx=0).stop 2024-05-09T05:01:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.229+0000 7f48d1b19640 1 -- 172.21.15.60:0/4997976 shutdown_connections 2024-05-09T05:01:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.229+0000 7f48d1b19640 1 --2- 172.21.15.60:0/4997976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48cc15f3e0 0x7f48cc0b4630 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.229+0000 7f48d1b19640 1 -- 172.21.15.60:0/4997976 >> 172.21.15.60:0/4997976 conn(0x7f48cc0d1470 msgr2=0x7f48cc0d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.230+0000 7f48d1b19640 1 -- 172.21.15.60:0/4997976 shutdown_connections 2024-05-09T05:01:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.230+0000 7f48d1b19640 1 -- 172.21.15.60:0/4997976 wait complete. 2024-05-09T05:01:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.231+0000 7f48d1b19640 1 Processor -- start 2024-05-09T05:01:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.231+0000 7f48d1b19640 1 -- start start 2024-05-09T05:01:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48d1b19640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48cc073310 0x7f48cc0c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48d1b19640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48cc09c8b0 0x7f48cc0c8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48d1b19640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48cc15f3e0 0x7f48cc09a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48d1b19640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc072ab0 con 0x7f48cc15f3e0 2024-05-09T05:01:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48d1b19640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f48cc0727d0 con 0x7f48cc073310 2024-05-09T05:01:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48d1b19640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f48cc15f7c0 con 0x7f48cc09c8b0 2024-05-09T05:01:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48cbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48cc15f3e0 0x7f48cc09a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48cbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48cc15f3e0 0x7f48cc09a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59894/0 (socket says 172.21.15.60:59894) 2024-05-09T05:01:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48cbfff640 1 -- 172.21.15.60:0/1476026785 learned_addr learned my addr 172.21.15.60:0/1476026785 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48cb7fe640 1 --2- 172.21.15.60:0/1476026785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48cc073310 0x7f48cc0c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48caffd640 1 --2- 172.21.15.60:0/1476026785 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48cc09c8b0 0x7f48cc0c8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48cbfff640 1 -- 172.21.15.60:0/1476026785 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48cc09c8b0 msgr2=0x7f48cc0c8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48cbfff640 1 --2- 172.21.15.60:0/1476026785 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48cc09c8b0 0x7f48cc0c8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48cbfff640 1 -- 172.21.15.60:0/1476026785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48cc073310 msgr2=0x7f48cc0c9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48cbfff640 1 --2- 172.21.15.60:0/1476026785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48cc073310 0x7f48cc0c9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48cbfff640 1 -- 172.21.15.60:0/1476026785 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48cc0c8bf0 con 0x7f48cc15f3e0 2024-05-09T05:01:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48caffd640 1 --2- 172.21.15.60:0/1476026785 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48cc09c8b0 0x7f48cc0c8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48cb7fe640 1 --2- 172.21.15.60:0/1476026785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48cc073310 0x7f48cc0c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48cbfff640 1 --2- 172.21.15.60:0/1476026785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48cc15f3e0 0x7f48cc09a820 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f48c400cb10 tx=0x7f48c400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48b3fff640 1 -- 172.21.15.60:0/1476026785 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c4017070 con 0x7f48cc15f3e0 2024-05-09T05:01:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48b3fff640 1 -- 172.21.15.60:0/1476026785 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48c400ac10 con 0x7f48cc15f3e0 2024-05-09T05:01:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48d1b19640 1 -- 172.21.15.60:0/1476026785 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48cc0c8ed0 con 0x7f48cc15f3e0 2024-05-09T05:01:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48b3fff640 1 -- 172.21.15.60:0/1476026785 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c4012400 con 0x7f48cc15f3e0 2024-05-09T05:01:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.232+0000 7f48d1b19640 1 -- 172.21.15.60:0/1476026785 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48cc0c9490 con 0x7f48cc15f3e0 2024-05-09T05:01:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.234+0000 7f48d1b19640 1 -- 172.21.15.60:0/1476026785 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48cc051040 con 0x7f48cc15f3e0 2024-05-09T05:01:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.234+0000 7f48b3fff640 1 -- 172.21.15.60:0/1476026785 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f48c401a070 con 0x7f48cc15f3e0 2024-05-09T05:01:11.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.235+0000 7f48b3fff640 1 --2- 172.21.15.60:0/1476026785 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f48b80762d0 0x7f48b8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.235+0000 7f48b3fff640 1 -- 172.21.15.60:0/1476026785 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7f48c4097530 con 0x7f48cc15f3e0 2024-05-09T05:01:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.235+0000 7f48cb7fe640 1 --2- 172.21.15.60:0/1476026785 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f48b80762d0 0x7f48b8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.235+0000 7f48cb7fe640 1 --2- 172.21.15.60:0/1476026785 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f48b80762d0 0x7f48b8078790 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f48bc034000 tx=0x7f48bc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.239+0000 7f48b3fff640 1 -- 172.21.15.60:0/1476026785 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f48c4060580 con 0x7f48cc15f3e0 2024-05-09T05:01:11.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.470+0000 7f48d1b19640 1 -- 172.21.15.60:0/1476026785 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f48cc0994b0 con 0x7f48cc15f3e0 2024-05-09T05:01:11.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.470+0000 7f48b3fff640 1 -- 172.21.15.60:0/1476026785 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7f48c4064880 con 0x7f48cc15f3e0 2024-05-09T05:01:11.471 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T05:01:11.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.472+0000 7f48d1b19640 1 -- 172.21.15.60:0/1476026785 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f48b80762d0 msgr2=0x7f48b8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:11.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.472+0000 7f48d1b19640 1 --2- 172.21.15.60:0/1476026785 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f48b80762d0 0x7f48b8078790 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f48bc034000 tx=0x7f48bc02e000 comp rx=0 tx=0).stop 2024-05-09T05:01:11.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.472+0000 7f48d1b19640 1 -- 172.21.15.60:0/1476026785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48cc15f3e0 msgr2=0x7f48cc09a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:11.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.472+0000 7f48d1b19640 1 --2- 172.21.15.60:0/1476026785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48cc15f3e0 0x7f48cc09a820 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f48c400cb10 tx=0x7f48c400a060 comp rx=0 tx=0).stop 2024-05-09T05:01:11.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.472+0000 7f48d1b19640 1 -- 172.21.15.60:0/1476026785 shutdown_connections 2024-05-09T05:01:11.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.472+0000 7f48d1b19640 1 --2- 172.21.15.60:0/1476026785 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f48b80762d0 0x7f48b8078790 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.472+0000 7f48d1b19640 1 --2- 172.21.15.60:0/1476026785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48cc15f3e0 0x7f48cc09a820 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.472+0000 7f48d1b19640 1 --2- 172.21.15.60:0/1476026785 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48cc09c8b0 0x7f48cc0c8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.472+0000 7f48d1b19640 1 --2- 172.21.15.60:0/1476026785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48cc073310 0x7f48cc0c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:11.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.472+0000 7f48d1b19640 1 -- 172.21.15.60:0/1476026785 >> 172.21.15.60:0/1476026785 conn(0x7f48cc0d1470 msgr2=0x7f48cc15fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:11.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.472+0000 7f48d1b19640 1 -- 172.21.15.60:0/1476026785 shutdown_connections 2024-05-09T05:01:11.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11.472+0000 7f48d1b19640 1 -- 172.21.15.60:0/1476026785 wait complete. 2024-05-09T05:01:11.483 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-09T05:01:11.483 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T05:01:11.483 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2024-05-09T05:01:11.484 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T05:01:11.484 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T05:01:11.484 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-09T05:01:11.484 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-05-09T05:01:11.484 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T05:01:11.484 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T05:01:11.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:11.484 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2024-05-09T05:01:11.501 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T05:01:11.501 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T05:01:11.501 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T05:01:11.501 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:01:11.501 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2024-05-09T05:01:11.501 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:01:11.501 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:01:11.501 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2024-05-09T05:01:11.501 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2024-05-09T05:01:11.502 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:01:11.502 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2024-05-09T05:01:11.502 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2024-05-09T05:01:11.502 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T05:01:11.502 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:01:11.502 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2024-05-09T05:01:11.502 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T05:01:11.502 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 42692sec preferred_lft 42692sec 2024-05-09T05:01:11.502 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2024-05-09T05:01:11.502 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:01:11.502 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:01:11.502 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:01:11.502 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2024-05-09T05:01:11.503 INFO:teuthology.orchestra.run.smithi060.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T05:01:11.503 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 42:54:29:48:e0:84 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:01:11.503 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T05:01:11.503 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:01:11.503 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::4889:3fff:fe4f:b9fb/64 scope link 2024-05-09T05:01:11.503 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:01:11.503 INFO:teuthology.orchestra.run.smithi060.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T05:01:11.503 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 42:54:29:48:e0:84 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T05:01:11.503 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::4054:29ff:fe48:e084/64 scope link 2024-05-09T05:01:11.503 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:01:11.503 INFO:teuthology.orchestra.run.smithi060.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T05:01:11.503 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 72:cf:f2:a5:7d:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T05:01:11.503 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::70cf:f2ff:fea5:7dba/64 scope link 2024-05-09T05:01:11.503 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:01:11.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:11.504 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T05:01:11.566 INFO:teuthology.orchestra.run.smithi060.stdout:1 2024-05-09T05:01:11.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:11.568 DEBUG:teuthology.orchestra.run.smithi060:> route 2024-05-09T05:01:11.624 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2024-05-09T05:01:11.624 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T05:01:11.624 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T05:01:11.625 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T05:01:11.625 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T05:01:11.625 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:01:11.625 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-05-09T05:01:11.625 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T05:01:11.625 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T05:01:11.625 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T05:01:11.625 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-05-09T05:01:11.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:01:11.748 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:01:11.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:11.755 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2024-05-09T05:01:11.771 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T05:01:11.771 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T05:01:11.772 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T05:01:11.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:11.772 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2024-05-09T05:01:11.838 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T05:01:11.839 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T05:01:11.894 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T05:01:11.894 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T05:01:11.895 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T05:01:11.950 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-05-09T05:01:11.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:11.950 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T05:01:12.035 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T05:01:12.035 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T05:01:12.046 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T05:01:12.046 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T05:01:12.046 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.114:3300,172.21.15.114:3301" 2024-05-09T05:01:12.046 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2.cephfs=/ 2024-05-09T05:01:12.047 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.60:3300/172.21.15.114:3300/172.21.15.114:3301" will pass to kernel 2024-05-09T05:01:12.049 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-09T05:01:12.049 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2024-05-09T05:01:12.049 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-09T05:01:12.049 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2024-05-09T05:01:12.050 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-09T05:01:12.050 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-09T05:01:12.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:12.050 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T05:01:12.078 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T05:01:12.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T05:01:12.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:01:12.181 INFO:teuthology.orchestra.run.smithi060.stdout:client5058 2024-05-09T05:01:12.181 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:01:12.181 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:01:12.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_symlink_recovery false 2024-05-09T05:01:12.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.345+0000 7f6b743c8640 1 Processor -- start 2024-05-09T05:01:12.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.345+0000 7f6b743c8640 1 -- start start 2024-05-09T05:01:12.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.346+0000 7f6b743c8640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6b6c15f3d0 0x7f6b6c0cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:12.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.346+0000 7f6b743c8640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b6c0cea00 con 0x7f6b6c0d1a60 2024-05-09T05:01:12.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.346+0000 7f6b743c8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b6c0d0ca0 con 0x7f6b6c072840 2024-05-09T05:01:12.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.346+0000 7f6b743c8640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f6b6c0d15d0 con 0x7f6b6c15f3d0 2024-05-09T05:01:12.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.346+0000 7f6b71963640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6b6c0d1a60 0x7f6b6c072130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33548/0 (socket says 172.21.15.60:33548) 2024-05-09T05:01:12.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.346+0000 7f6b71963640 1 -- 172.21.15.60:0/3991934099 learned_addr learned my addr 172.21.15.60:0/3991934099 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:12.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b72164640 1 --2- 172.21.15.60:0/3991934099 >> v2:172.21.15.114:3300/0 conn(0x7f6b6c15f3d0 0x7f6b6c0cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:12.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b71162640 1 -- 172.21.15.60:0/3991934099 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3854166130 0 0) 0x7f6b6c0cea00 con 0x7f6b6c0d1a60 2024-05-09T05:01:12.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b71162640 1 -- 172.21.15.60:0/3991934099 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b54003520 con 0x7f6b6c0d1a60 2024-05-09T05:01:12.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b71162640 1 -- 172.21.15.60:0/3991934099 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3776295551 0 0) 0x7f6b6c0d0ca0 con 0x7f6b6c072840 2024-05-09T05:01:12.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b71162640 1 -- 172.21.15.60:0/3991934099 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b6c0cea00 con 0x7f6b6c072840 2024-05-09T05:01:12.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b71162640 1 -- 172.21.15.60:0/3991934099 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1105001623 0 0) 0x7f6b54003520 con 0x7f6b6c0d1a60 2024-05-09T05:01:12.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b71162640 1 -- 172.21.15.60:0/3991934099 >> v2:172.21.15.114:3300/0 conn(0x7f6b6c15f3d0 msgr2=0x7f6b6c0cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:12.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b71162640 1 --2- 172.21.15.60:0/3991934099 >> v2:172.21.15.114:3300/0 conn(0x7f6b6c15f3d0 0x7f6b6c0cdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:12.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b71162640 1 -- 172.21.15.60:0/3991934099 >> v1:172.21.15.114:6789/0 conn(0x7f6b6c072840 legacy=0x7f6b6c0cee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:12.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b71162640 1 -- 172.21.15.60:0/3991934099 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b6c0d0f80 con 0x7f6b6c0d1a60 2024-05-09T05:01:12.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b71162640 1 -- 172.21.15.60:0/3991934099 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f6b600016f0 con 0x7f6b6c0d1a60 2024-05-09T05:01:12.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b72164640 1 --2- 172.21.15.60:0/3991934099 >> v2:172.21.15.114:3300/0 conn(0x7f6b6c15f3d0 0x7f6b6c0cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:01:12.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b71162640 1 -- 172.21.15.60:0/3991934099 >> v1:172.21.15.60:6789/0 conn(0x7f6b6c0d1a60 legacy=0x7f6b6c072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:12.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b71162640 1 --2- 172.21.15.60:0/3991934099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b54003c30 0x7f6b54004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:12.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b71162640 1 --2- 172.21.15.60:0/3991934099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b54004800 0x7f6b54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:12.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b71162640 1 --2- 172.21.15.60:0/3991934099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b54005210 0x7f6b5400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:12.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b71162640 1 -- 172.21.15.60:0/3991934099 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6c0d15d0 con 0x7f6b54003c30 2024-05-09T05:01:12.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b71162640 1 -- 172.21.15.60:0/3991934099 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5400c140 con 0x7f6b54004800 2024-05-09T05:01:12.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b71162640 1 -- 172.21.15.60:0/3991934099 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6b5400c340 con 0x7f6b54005210 2024-05-09T05:01:12.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b72164640 1 --2- 172.21.15.60:0/3991934099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b54003c30 0x7f6b54004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:12.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b71963640 1 --2- 172.21.15.60:0/3991934099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b54004800 0x7f6b54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:12.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b72965640 1 --2- 172.21.15.60:0/3991934099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b54005210 0x7f6b5400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:12.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b72164640 1 -- 172.21.15.60:0/3991934099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b54005210 msgr2=0x7f6b5400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:12.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b72164640 1 --2- 172.21.15.60:0/3991934099 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b54005210 0x7f6b5400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:12.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b72164640 1 -- 172.21.15.60:0/3991934099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b54004800 msgr2=0x7f6b54004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:12.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b72164640 1 --2- 172.21.15.60:0/3991934099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b54004800 0x7f6b54004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:12.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.347+0000 7f6b72164640 1 -- 172.21.15.60:0/3991934099 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b54003520 con 0x7f6b54003c30 2024-05-09T05:01:12.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.348+0000 7f6b72965640 1 -- 172.21.15.60:0/3991934099 reap_dead start 2024-05-09T05:01:12.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.348+0000 7f6b72164640 1 --2- 172.21.15.60:0/3991934099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b54003c30 0x7f6b54004030 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c009c90 tx=0x7f6b54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddc86d3b2bd879af server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:12.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.348+0000 7f6b71162640 1 -- 172.21.15.60:0/3991934099 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c036070 con 0x7f6b54003c30 2024-05-09T05:01:12.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.348+0000 7f6b71162640 1 -- 172.21.15.60:0/3991934099 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b5c031400 con 0x7f6b54003c30 2024-05-09T05:01:12.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.348+0000 7f6b71162640 1 -- 172.21.15.60:0/3991934099 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c009860 con 0x7f6b54003c30 2024-05-09T05:01:12.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.349+0000 7f6b743c8640 1 -- 172.21.15.60:0/3991934099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b54003c30 msgr2=0x7f6b54004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:12.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.349+0000 7f6b743c8640 1 --2- 172.21.15.60:0/3991934099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b54003c30 0x7f6b54004030 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c009c90 tx=0x7f6b54005830 comp rx=0 tx=0).stop 2024-05-09T05:01:12.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.349+0000 7f6b743c8640 1 -- 172.21.15.60:0/3991934099 shutdown_connections 2024-05-09T05:01:12.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.349+0000 7f6b743c8640 1 --2- 172.21.15.60:0/3991934099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b54003c30 0x7f6b54004030 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:12.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.349+0000 7f6b743c8640 1 -- 172.21.15.60:0/3991934099 >> 172.21.15.60:0/3991934099 conn(0x7f6b6c073420 msgr2=0x7f6b6c0b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:12.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.349+0000 7f6b743c8640 1 -- 172.21.15.60:0/3991934099 shutdown_connections 2024-05-09T05:01:12.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.349+0000 7f6b743c8640 1 -- 172.21.15.60:0/3991934099 wait complete. 2024-05-09T05:01:12.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.350+0000 7f6b743c8640 1 Processor -- start 2024-05-09T05:01:12.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.350+0000 7f6b743c8640 1 -- start start 2024-05-09T05:01:12.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.351+0000 7f6b743c8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b54003c30 0x7f6b6c15c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:12.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.351+0000 7f6b743c8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b6c15e110 0x7f6b6c15a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:12.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.351+0000 7f6b743c8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b6c143150 0x7f6b6c1576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:12.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.351+0000 7f6b743c8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6c15f7b0 con 0x7f6b6c143150 2024-05-09T05:01:12.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.351+0000 7f6b743c8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6c163eb0 con 0x7f6b6c15e110 2024-05-09T05:01:12.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.351+0000 7f6b743c8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6b6c15f930 con 0x7f6b54003c30 2024-05-09T05:01:12.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.351+0000 7f6b72965640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b6c143150 0x7f6b6c1576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:12.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.351+0000 7f6b72965640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b6c143150 0x7f6b6c1576d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59914/0 (socket says 172.21.15.60:59914) 2024-05-09T05:01:12.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.351+0000 7f6b72965640 1 -- 172.21.15.60:0/2220097341 learned_addr learned my addr 172.21.15.60:0/2220097341 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:12.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.351+0000 7f6b71963640 1 --2- 172.21.15.60:0/2220097341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b6c15e110 0x7f6b6c15a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:12.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.351+0000 7f6b72164640 1 --2- 172.21.15.60:0/2220097341 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b54003c30 0x7f6b6c15c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:12.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.351+0000 7f6b72965640 1 -- 172.21.15.60:0/2220097341 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b54003c30 msgr2=0x7f6b6c15c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:12.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.351+0000 7f6b72965640 1 --2- 172.21.15.60:0/2220097341 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b54003c30 0x7f6b6c15c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:12.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.351+0000 7f6b72965640 1 -- 172.21.15.60:0/2220097341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b6c15e110 msgr2=0x7f6b6c15a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:12.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.351+0000 7f6b72965640 1 --2- 172.21.15.60:0/2220097341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b6c15e110 0x7f6b6c15a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:12.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.351+0000 7f6b72965640 1 -- 172.21.15.60:0/2220097341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b6c15d070 con 0x7f6b6c143150 2024-05-09T05:01:12.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.352+0000 7f6b72965640 1 --2- 172.21.15.60:0/2220097341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b6c143150 0x7f6b6c1576d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c0d20e0 tx=0x7f6b6400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:12.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.352+0000 7f6b6affd640 1 -- 172.21.15.60:0/2220097341 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b64003970 con 0x7f6b6c143150 2024-05-09T05:01:12.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.352+0000 7f6b6affd640 1 -- 172.21.15.60:0/2220097341 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b64018070 con 0x7f6b6c143150 2024-05-09T05:01:12.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.352+0000 7f6b6affd640 1 -- 172.21.15.60:0/2220097341 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b64003d50 con 0x7f6b6c143150 2024-05-09T05:01:12.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.352+0000 7f6b743c8640 1 -- 172.21.15.60:0/2220097341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b6c15d350 con 0x7f6b6c143150 2024-05-09T05:01:12.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.352+0000 7f6b743c8640 1 -- 172.21.15.60:0/2220097341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b6c15d910 con 0x7f6b6c143150 2024-05-09T05:01:12.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.353+0000 7f6b743c8640 1 -- 172.21.15.60:0/2220097341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b6c051040 con 0x7f6b6c143150 2024-05-09T05:01:12.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.353+0000 7f6b6affd640 1 -- 172.21.15.60:0/2220097341 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6b640050d0 con 0x7f6b6c143150 2024-05-09T05:01:12.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.354+0000 7f6b6affd640 1 --2- 172.21.15.60:0/2220097341 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6b580762d0 0x7f6b58078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:12.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.354+0000 7f6b6affd640 1 -- 172.21.15.60:0/2220097341 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7f6b64097470 con 0x7f6b6c143150 2024-05-09T05:01:12.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.354+0000 7f6b72164640 1 --2- 172.21.15.60:0/2220097341 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6b580762d0 0x7f6b58078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:12.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.355+0000 7f6b72164640 1 --2- 172.21.15.60:0/2220097341 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6b580762d0 0x7f6b58078790 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c033000 tx=0x7f6b5c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:12.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.358+0000 7f6b6affd640 1 -- 172.21.15.60:0/2220097341 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6b64060ba0 con 0x7f6b6c143150 2024-05-09T05:01:12.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.550+0000 7f6b743c8640 1 -- 172.21.15.60:0/2220097341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mds_symlink_recovery}] v 0) -- 0x7f6b6c0b8220 con 0x7f6b6c143150 2024-05-09T05:01:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.553+0000 7f6b6affd640 1 -- 172.21.15.60:0/2220097341 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_symlink_recovery}]=0 v11)=0 v11) ==== 122+0+0 (secure 0 0 0) 0x7f6b640604a0 con 0x7f6b6c143150 2024-05-09T05:01:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.555+0000 7f6b743c8640 1 -- 172.21.15.60:0/2220097341 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6b580762d0 msgr2=0x7f6b58078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.555+0000 7f6b743c8640 1 --2- 172.21.15.60:0/2220097341 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6b580762d0 0x7f6b58078790 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c033000 tx=0x7f6b5c008030 comp rx=0 tx=0).stop 2024-05-09T05:01:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.555+0000 7f6b743c8640 1 -- 172.21.15.60:0/2220097341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b6c143150 msgr2=0x7f6b6c1576d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.555+0000 7f6b743c8640 1 --2- 172.21.15.60:0/2220097341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b6c143150 0x7f6b6c1576d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c0d20e0 tx=0x7f6b6400eb20 comp rx=0 tx=0).stop 2024-05-09T05:01:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.555+0000 7f6b743c8640 1 -- 172.21.15.60:0/2220097341 shutdown_connections 2024-05-09T05:01:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.555+0000 7f6b743c8640 1 --2- 172.21.15.60:0/2220097341 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6b580762d0 0x7f6b58078790 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.555+0000 7f6b743c8640 1 --2- 172.21.15.60:0/2220097341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b6c143150 0x7f6b6c1576d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.555+0000 7f6b743c8640 1 --2- 172.21.15.60:0/2220097341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b6c15e110 0x7f6b6c15a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.555+0000 7f6b743c8640 1 --2- 172.21.15.60:0/2220097341 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b54003c30 0x7f6b6c15c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.555+0000 7f6b743c8640 1 -- 172.21.15.60:0/2220097341 >> 172.21.15.60:0/2220097341 conn(0x7f6b6c073420 msgr2=0x7f6b6c0b72c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.556+0000 7f6b743c8640 1 -- 172.21.15.60:0/2220097341 shutdown_connections 2024-05-09T05:01:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.556+0000 7f6b743c8640 1 -- 172.21.15.60:0/2220097341 wait complete. 2024-05-09T05:01:12.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:12.568 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir sym_dir0) 2024-05-09T05:01:12.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:12.589 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch sym_dir0/file1) 2024-05-09T05:01:12.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:12.647 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln -s sym_dir0/file1 sym_dir0/symlink_file1) 2024-05-09T05:01:12.704 DEBUG:teuthology.orchestra.run.smithi060:> stdin-killer -- python3 -c ' 2024-05-09T05:01:12.705 DEBUG:teuthology.orchestra.run.smithi060:> import os 2024-05-09T05:01:12.705 DEBUG:teuthology.orchestra.run.smithi060:> import stat 2024-05-09T05:01:12.705 DEBUG:teuthology.orchestra.run.smithi060:> 2024-05-09T05:01:12.705 DEBUG:teuthology.orchestra.run.smithi060:> print(os.lstat("/home/ubuntu/cephtest/mnt.0/sym_dir0/symlink_file1").st_ino) 2024-05-09T05:01:12.705 DEBUG:teuthology.orchestra.run.smithi060:> ' 2024-05-09T05:01:12.813 INFO:teuthology.orchestra.run.smithi060.stdout:1099511628278 2024-05-09T05:01:12.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:01:12.825 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:12.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.974+0000 7feb2bf2b640 1 Processor -- start 2024-05-09T05:01:12.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.974+0000 7feb2bf2b640 1 -- start start 2024-05-09T05:01:12.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.975+0000 7feb2bf2b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb2415c3a0 0x7feb24070150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:12.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.975+0000 7feb2bf2b640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7feb24073c30 0x7feb24070690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:12.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.975+0000 7feb2bf2b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb24071b80 con 0x7feb240ac430 2024-05-09T05:01:12.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.975+0000 7feb2bf2b640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7feb24071e60 con 0x7feb24073c30 2024-05-09T05:01:12.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.975+0000 7feb2bf2b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7feb24072140 con 0x7feb2415c3a0 2024-05-09T05:01:12.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.975+0000 7feb294c6640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7feb240ac430 0x7feb24099a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33558/0 (socket says 172.21.15.60:33558) 2024-05-09T05:01:12.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.975+0000 7feb294c6640 1 -- 172.21.15.60:0/2750897671 learned_addr learned my addr 172.21.15.60:0/2750897671 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:12.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.976+0000 7feb29cc7640 1 --2- 172.21.15.60:0/2750897671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb2415c3a0 0x7feb24070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:12.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.976+0000 7feb2a4c8640 1 --2- 172.21.15.60:0/2750897671 >> v2:172.21.15.114:3300/0 conn(0x7feb24073c30 0x7feb24070690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:12.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.976+0000 7feb28cc5640 1 -- 172.21.15.60:0/2750897671 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3144376632 0 0) 0x7feb24071b80 con 0x7feb240ac430 2024-05-09T05:01:12.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.976+0000 7feb28cc5640 1 -- 172.21.15.60:0/2750897671 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb10003520 con 0x7feb240ac430 2024-05-09T05:01:12.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.976+0000 7feb29cc7640 1 -- 172.21.15.60:0/2750897671 >> v2:172.21.15.114:3300/0 conn(0x7feb24073c30 msgr2=0x7feb24070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:12.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.976+0000 7feb29cc7640 1 --2- 172.21.15.60:0/2750897671 >> v2:172.21.15.114:3300/0 conn(0x7feb24073c30 0x7feb24070690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:12.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.976+0000 7feb29cc7640 1 -- 172.21.15.60:0/2750897671 >> v1:172.21.15.60:6789/0 conn(0x7feb240ac430 legacy=0x7feb24099a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:12.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.976+0000 7feb29cc7640 1 -- 172.21.15.60:0/2750897671 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb24072420 con 0x7feb2415c3a0 2024-05-09T05:01:12.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.977+0000 7feb29cc7640 1 --2- 172.21.15.60:0/2750897671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb2415c3a0 0x7feb24070150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feb180097d0 tx=0x7feb1802a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb170655dce9aa8d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:12.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.977+0000 7feb28cc5640 1 -- 172.21.15.60:0/2750897671 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb18037070 con 0x7feb2415c3a0 2024-05-09T05:01:12.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.977+0000 7feb28cc5640 1 -- 172.21.15.60:0/2750897671 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb18032440 con 0x7feb2415c3a0 2024-05-09T05:01:12.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.977+0000 7feb28cc5640 1 -- 172.21.15.60:0/2750897671 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb18032810 con 0x7feb2415c3a0 2024-05-09T05:01:12.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.978+0000 7feb2bf2b640 1 -- 172.21.15.60:0/2750897671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb2415c3a0 msgr2=0x7feb24070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:12.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.978+0000 7feb2bf2b640 1 --2- 172.21.15.60:0/2750897671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb2415c3a0 0x7feb24070150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feb180097d0 tx=0x7feb1802a1c0 comp rx=0 tx=0).stop 2024-05-09T05:01:12.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.978+0000 7feb2bf2b640 1 -- 172.21.15.60:0/2750897671 shutdown_connections 2024-05-09T05:01:12.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.978+0000 7feb2bf2b640 1 --2- 172.21.15.60:0/2750897671 >> v2:172.21.15.114:3300/0 conn(0x7feb24073c30 0x7feb24070690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:12.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.978+0000 7feb2bf2b640 1 --2- 172.21.15.60:0/2750897671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb2415c3a0 0x7feb24070150 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:12.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.978+0000 7feb2bf2b640 1 -- 172.21.15.60:0/2750897671 >> 172.21.15.60:0/2750897671 conn(0x7feb240972f0 msgr2=0x7feb240976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:12.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.978+0000 7feb2bf2b640 1 -- 172.21.15.60:0/2750897671 shutdown_connections 2024-05-09T05:01:12.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.978+0000 7feb2bf2b640 1 -- 172.21.15.60:0/2750897671 wait complete. 2024-05-09T05:01:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.979+0000 7feb2bf2b640 1 Processor -- start 2024-05-09T05:01:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.979+0000 7feb2bf2b640 1 -- start start 2024-05-09T05:01:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.980+0000 7feb2bf2b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb24073c30 0x7feb24096790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.980+0000 7feb2bf2b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb240ac430 0x7feb240af600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.980+0000 7feb2bf2b640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb2415c3a0 0x7feb240b1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.980+0000 7feb2bf2b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7feb24071810 con 0x7feb24073c30 2024-05-09T05:01:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.980+0000 7feb2bf2b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7feb2415c780 con 0x7feb240ac430 2024-05-09T05:01:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.980+0000 7feb2bf2b640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7feb24097de0 con 0x7feb2415c3a0 2024-05-09T05:01:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.980+0000 7feb29cc7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb24073c30 0x7feb24096790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.980+0000 7feb29cc7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb24073c30 0x7feb24096790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59922/0 (socket says 172.21.15.60:59922) 2024-05-09T05:01:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.980+0000 7feb29cc7640 1 -- 172.21.15.60:0/3147069193 learned_addr learned my addr 172.21.15.60:0/3147069193 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:12.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.980+0000 7feb2a4c8640 1 --2- 172.21.15.60:0/3147069193 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb2415c3a0 0x7feb240b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:12.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.980+0000 7feb294c6640 1 --2- 172.21.15.60:0/3147069193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb240ac430 0x7feb240af600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:12.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.980+0000 7feb29cc7640 1 -- 172.21.15.60:0/3147069193 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb2415c3a0 msgr2=0x7feb240b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:12.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.980+0000 7feb29cc7640 1 --2- 172.21.15.60:0/3147069193 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb2415c3a0 0x7feb240b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:12.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.980+0000 7feb29cc7640 1 -- 172.21.15.60:0/3147069193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb240ac430 msgr2=0x7feb240af600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:12.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.980+0000 7feb29cc7640 1 --2- 172.21.15.60:0/3147069193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb240ac430 0x7feb240af600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:12.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.980+0000 7feb29cc7640 1 -- 172.21.15.60:0/3147069193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb240b00c0 con 0x7feb24073c30 2024-05-09T05:01:12.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.981+0000 7feb29cc7640 1 --2- 172.21.15.60:0/3147069193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb24073c30 0x7feb24096790 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7feb18035040 tx=0x7feb180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:12.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.981+0000 7feb167fc640 1 -- 172.21.15.60:0/3147069193 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb18037050 con 0x7feb24073c30 2024-05-09T05:01:12.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.981+0000 7feb167fc640 1 -- 172.21.15.60:0/3147069193 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb18038040 con 0x7feb24073c30 2024-05-09T05:01:12.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.981+0000 7feb2bf2b640 1 -- 172.21.15.60:0/3147069193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb240b03a0 con 0x7feb24073c30 2024-05-09T05:01:12.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.981+0000 7feb167fc640 1 -- 172.21.15.60:0/3147069193 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb18004470 con 0x7feb24073c30 2024-05-09T05:01:12.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.981+0000 7feb2bf2b640 1 -- 172.21.15.60:0/3147069193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb240b0960 con 0x7feb24073c30 2024-05-09T05:01:12.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.982+0000 7feb2bf2b640 1 -- 172.21.15.60:0/3147069193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb240ab9b0 con 0x7feb24073c30 2024-05-09T05:01:12.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.983+0000 7feb167fc640 1 -- 172.21.15.60:0/3147069193 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7feb18003610 con 0x7feb24073c30 2024-05-09T05:01:12.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.983+0000 7feb167fc640 1 --2- 172.21.15.60:0/3147069193 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7feb080762d0 0x7feb08078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:12.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.983+0000 7feb167fc640 1 -- 172.21.15.60:0/3147069193 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7feb180b8130 con 0x7feb24073c30 2024-05-09T05:01:12.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.983+0000 7feb294c6640 1 --2- 172.21.15.60:0/3147069193 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7feb080762d0 0x7feb08078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:12.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.984+0000 7feb294c6640 1 --2- 172.21.15.60:0/3147069193 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7feb080762d0 0x7feb08078790 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7feb20007a60 tx=0x7feb20009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:12.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:12.988+0000 7feb167fc640 1 -- 172.21.15.60:0/3147069193 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7feb240ab9b0 con 0x7feb24073c30 2024-05-09T05:01:13.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.218+0000 7feb2bf2b640 1 -- 172.21.15.60:0/3147069193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb240b0d90 con 0x7feb24073c30 2024-05-09T05:01:13.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.218+0000 7feb167fc640 1 -- 172.21.15.60:0/3147069193 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5181 (secure 0 0 0) 0x7feb18081910 con 0x7feb24073c30 2024-05-09T05:01:13.219 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:13.219 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4547,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2372252858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2372252858},{"type":"v1","addr":"172.21.15.114:6836","nonce":2372252858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1761282610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1761282610},{"type":"v1","addr":"172.21.15.114:6837","nonce":1761282610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4765,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/300408012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":300408012},{"type":"v1","addr":"172.21.15.60:6835","nonce":300408012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:04.698975+0000","modified":"2024-05-09T05:01:07.722167+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4893},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4893":{"gid":4893,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/399160914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":399160914},{"type":"v1","addr":"172.21.15.60:6834","nonce":399160914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4893,"qdb_cluster":[4893]},"id":2}]} 2024-05-09T05:01:13.219 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 17 2024-05-09T05:01:13.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.220+0000 7feb2bf2b640 1 -- 172.21.15.60:0/3147069193 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7feb080762d0 msgr2=0x7feb08078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:13.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.220+0000 7feb2bf2b640 1 --2- 172.21.15.60:0/3147069193 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7feb080762d0 0x7feb08078790 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7feb20007a60 tx=0x7feb20009040 comp rx=0 tx=0).stop 2024-05-09T05:01:13.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.220+0000 7feb2bf2b640 1 -- 172.21.15.60:0/3147069193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb24073c30 msgr2=0x7feb24096790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:13.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.220+0000 7feb2bf2b640 1 --2- 172.21.15.60:0/3147069193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb24073c30 0x7feb24096790 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7feb18035040 tx=0x7feb180046c0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.220+0000 7feb2bf2b640 1 -- 172.21.15.60:0/3147069193 shutdown_connections 2024-05-09T05:01:13.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.220+0000 7feb2bf2b640 1 --2- 172.21.15.60:0/3147069193 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7feb080762d0 0x7feb08078790 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.220+0000 7feb2bf2b640 1 --2- 172.21.15.60:0/3147069193 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7feb2415c3a0 0x7feb240b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.220+0000 7feb2bf2b640 1 --2- 172.21.15.60:0/3147069193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb240ac430 0x7feb240af600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.220+0000 7feb2bf2b640 1 --2- 172.21.15.60:0/3147069193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb24073c30 0x7feb24096790 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.220+0000 7feb2bf2b640 1 -- 172.21.15.60:0/3147069193 >> 172.21.15.60:0/3147069193 conn(0x7feb240972f0 msgr2=0x7feb2409a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:13.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.221+0000 7feb2bf2b640 1 -- 172.21.15.60:0/3147069193 shutdown_connections 2024-05-09T05:01:13.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.221+0000 7feb2bf2b640 1 -- 172.21.15.60:0/3147069193 wait complete. 2024-05-09T05:01:13.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-05-09T05:01:13.363 INFO:teuthology.orchestra.run.smithi060.stdout:{"message":"","return_code":0} 2024-05-09T05:01:13.364 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-05-09T05:01:13.364 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f6.00000000 symlink 2024-05-09T05:01:13.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.409+0000 7f8ab5a14f40 1 Processor -- start 2024-05-09T05:01:13.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.409+0000 7f8ab5a14f40 1 -- start start 2024-05-09T05:01:13.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.410+0000 7f8ab5a14f40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5564a202dfc0 0x5564a2031ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:13.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.410+0000 7f8ab5a14f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5564a2079340 0x5564a2107720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:13.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.410+0000 7f8ab5a14f40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564a20f0510 0x5564a20ee470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:13.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.410+0000 7f8ab5a14f40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5564a1fd6f50 con 0x5564a20f0510 2024-05-09T05:01:13.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.410+0000 7f8ab5a14f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5564a20797b0 con 0x5564a2079340 2024-05-09T05:01:13.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.410+0000 7f8ab5a14f40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x5564a205f480 con 0x5564a202dfc0 2024-05-09T05:01:13.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.410+0000 7f8ab30a0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564a20f0510 0x5564a20ee470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:13.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.410+0000 7f8ab30a0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564a20f0510 0x5564a20ee470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59932/0 (socket says 172.21.15.60:59932) 2024-05-09T05:01:13.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.410+0000 7f8ab30a0640 1 -- 172.21.15.60:0/3457546013 learned_addr learned my addr 172.21.15.60:0/3457546013 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:13.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.410+0000 7f8ab289f640 1 --2- 172.21.15.60:0/3457546013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5564a202dfc0 0x5564a2031ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:13.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.410+0000 7f8ab209e640 1 --2- 172.21.15.60:0/3457546013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5564a2079340 0x5564a2107720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:13.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.410+0000 7f8ab30a0640 1 -- 172.21.15.60:0/3457546013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5564a202dfc0 msgr2=0x5564a2031ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:13.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.410+0000 7f8ab30a0640 1 --2- 172.21.15.60:0/3457546013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5564a202dfc0 0x5564a2031ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.410+0000 7f8ab30a0640 1 -- 172.21.15.60:0/3457546013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5564a2079340 msgr2=0x5564a2107720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:13.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.410+0000 7f8ab30a0640 1 --2- 172.21.15.60:0/3457546013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5564a2079340 0x5564a2107720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.410+0000 7f8ab30a0640 1 -- 172.21.15.60:0/3457546013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564a20312f0 con 0x5564a20f0510 2024-05-09T05:01:13.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.411+0000 7f8ab30a0640 1 --2- 172.21.15.60:0/3457546013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564a20f0510 0x5564a20ee470 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8aac0016c0 tx=0x7f8aac001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f623a70f667107 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:13.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.411+0000 7f8ab109c640 1 -- 172.21.15.60:0/3457546013 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac016070 con 0x5564a20f0510 2024-05-09T05:01:13.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.411+0000 7f8ab109c640 1 -- 172.21.15.60:0/3457546013 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aac004520 con 0x5564a20f0510 2024-05-09T05:01:13.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.411+0000 7f8ab109c640 1 -- 172.21.15.60:0/3457546013 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac0048d0 con 0x5564a20f0510 2024-05-09T05:01:13.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.411+0000 7f8ab5a14f40 1 -- 172.21.15.60:0/3457546013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564a20f0510 msgr2=0x5564a20ee470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:13.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.411+0000 7f8ab5a14f40 1 --2- 172.21.15.60:0/3457546013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564a20f0510 0x5564a20ee470 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8aac0016c0 tx=0x7f8aac001b80 comp rx=0 tx=0).stop 2024-05-09T05:01:13.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.411+0000 7f8ab5a14f40 1 -- 172.21.15.60:0/3457546013 shutdown_connections 2024-05-09T05:01:13.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.411+0000 7f8ab5a14f40 1 --2- 172.21.15.60:0/3457546013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564a20f0510 0x5564a20ee470 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.411+0000 7f8ab5a14f40 1 --2- 172.21.15.60:0/3457546013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5564a2079340 0x5564a2107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.411+0000 7f8ab5a14f40 1 --2- 172.21.15.60:0/3457546013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5564a202dfc0 0x5564a2031ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.411+0000 7f8ab5a14f40 1 -- 172.21.15.60:0/3457546013 >> 172.21.15.60:0/3457546013 conn(0x5564a20598d0 msgr2=0x5564a205a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:13.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.411+0000 7f8ab5a14f40 1 -- 172.21.15.60:0/3457546013 shutdown_connections 2024-05-09T05:01:13.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.411+0000 7f8ab5a14f40 1 -- 172.21.15.60:0/3457546013 wait complete. 2024-05-09T05:01:13.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.412+0000 7f8ab5a14f40 1 Processor -- start 2024-05-09T05:01:13.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.412+0000 7f8ab5a14f40 1 -- start start 2024-05-09T05:01:13.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8ab5a14f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5564a202dfc0 0x5564a2059f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:13.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8ab5a14f40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5564a2079340 0x5564a21a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:13.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8ab5a14f40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564a20f0510 0x5564a21a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:13.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8ab5a14f40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5564a205f6b0 con 0x5564a20f0510 2024-05-09T05:01:13.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8ab5a14f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5564a202e840 con 0x5564a202dfc0 2024-05-09T05:01:13.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8ab5a14f40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x5564a205ac50 con 0x5564a2079340 2024-05-09T05:01:13.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8ab30a0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564a20f0510 0x5564a21a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:13.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8ab30a0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564a20f0510 0x5564a21a0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59946/0 (socket says 172.21.15.60:59946) 2024-05-09T05:01:13.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8ab30a0640 1 -- 172.21.15.60:0/805020121 learned_addr learned my addr 172.21.15.60:0/805020121 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:13.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8ab289f640 1 --2- 172.21.15.60:0/805020121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5564a202dfc0 0x5564a2059f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:13.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8ab209e640 1 --2- 172.21.15.60:0/805020121 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5564a2079340 0x5564a21a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:13.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8ab30a0640 1 -- 172.21.15.60:0/805020121 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5564a2079340 msgr2=0x5564a21a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:13.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8ab30a0640 1 --2- 172.21.15.60:0/805020121 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5564a2079340 0x5564a21a0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8ab30a0640 1 -- 172.21.15.60:0/805020121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5564a202dfc0 msgr2=0x5564a2059f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:13.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8ab30a0640 1 --2- 172.21.15.60:0/805020121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5564a202dfc0 0x5564a2059f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8ab30a0640 1 -- 172.21.15.60:0/805020121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564a2075d30 con 0x5564a20f0510 2024-05-09T05:01:13.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8ab30a0640 1 --2- 172.21.15.60:0/805020121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564a20f0510 0x5564a21a0e40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8aac0044f0 tx=0x7f8aac004460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:13.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8a9bfff640 1 -- 172.21.15.60:0/805020121 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac016070 con 0x5564a20f0510 2024-05-09T05:01:13.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8a9bfff640 1 -- 172.21.15.60:0/805020121 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aac017040 con 0x5564a20f0510 2024-05-09T05:01:13.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8ab5a14f40 1 -- 172.21.15.60:0/805020121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5564a20742d0 con 0x5564a20f0510 2024-05-09T05:01:13.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.413+0000 7f8a9bfff640 1 -- 172.21.15.60:0/805020121 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac01d8c0 con 0x5564a20f0510 2024-05-09T05:01:13.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.414+0000 7f8ab5a14f40 1 -- 172.21.15.60:0/805020121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564a2106330 con 0x5564a20f0510 2024-05-09T05:01:13.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.415+0000 7f8a9bfff640 1 -- 172.21.15.60:0/805020121 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8aac01da60 con 0x5564a20f0510 2024-05-09T05:01:13.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.416+0000 7f8a9bfff640 1 --2- 172.21.15.60:0/805020121 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8aa0074cb0 0x7f8aa0077170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:13.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.416+0000 7f8a9bfff640 1 -- 172.21.15.60:0/805020121 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7f8aac097a50 con 0x5564a20f0510 2024-05-09T05:01:13.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.416+0000 7f8ab5a14f40 1 --2- 172.21.15.60:0/805020121 >> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] conn(0x5564a205a470 0x5564a20ea0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:13.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.416+0000 7f8ab5a14f40 1 -- 172.21.15.60:0/805020121 --> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] -- osd_op(unknown.0.0:1 6.2a 6:57fe7535:::100000001f6.00000000:head [getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) -- 0x5564a2105de0 con 0x5564a205a470 2024-05-09T05:01:13.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.416+0000 7f8ab289f640 1 --2- 172.21.15.60:0/805020121 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8aa0074cb0 0x7f8aa0077170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:13.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.416+0000 7f8ab209e640 1 --2- 172.21.15.60:0/805020121 >> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] conn(0x5564a205a470 0x5564a20ea0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:13.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.417+0000 7f8ab289f640 1 --2- 172.21.15.60:0/805020121 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8aa0074cb0 0x7f8aa0077170 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4005400 tx=0x7f8aa400c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:13.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.417+0000 7f8ab209e640 1 --2- 172.21.15.60:0/805020121 >> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] conn(0x5564a205a470 0x5564a20ea0f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:13.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.417+0000 7f8ab209e640 1 -- 172.21.15.60:0/805020121 <== osd.4 v2:172.21.15.114:6801/3462725972 1 ==== osd_op_reply(1 100000001f6.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) ==== 164+0+0 (crc 0 0 0) 0x7f8aa8055520 con 0x5564a205a470 2024-05-09T05:01:13.420 INFO:teuthology.orchestra.run.smithi060.stderr:error getting xattr cephfs_data/100000001f6.00000000/symlink: (61) No data available 2024-05-09T05:01:13.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.417+0000 7f8ab5a14f40 1 -- 172.21.15.60:0/805020121 >> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] conn(0x5564a205a470 msgr2=0x5564a20ea0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:13.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.417+0000 7f8ab5a14f40 1 --2- 172.21.15.60:0/805020121 >> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] conn(0x5564a205a470 0x5564a20ea0f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.417+0000 7f8ab5a14f40 1 -- 172.21.15.60:0/805020121 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8aa0074cb0 msgr2=0x7f8aa0077170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:13.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.417+0000 7f8ab5a14f40 1 --2- 172.21.15.60:0/805020121 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8aa0074cb0 0x7f8aa0077170 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4005400 tx=0x7f8aa400c040 comp rx=0 tx=0).stop 2024-05-09T05:01:13.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.417+0000 7f8ab5a14f40 1 -- 172.21.15.60:0/805020121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564a20f0510 msgr2=0x5564a21a0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:13.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.417+0000 7f8ab5a14f40 1 --2- 172.21.15.60:0/805020121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5564a20f0510 0x5564a21a0e40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8aac0044f0 tx=0x7f8aac004460 comp rx=0 tx=0).stop 2024-05-09T05:01:13.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.417+0000 7f8ab30a0640 1 -- 172.21.15.60:0/805020121 reap_dead start 2024-05-09T05:01:13.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.417+0000 7f8ab5a14f40 1 -- 172.21.15.60:0/805020121 shutdown_connections 2024-05-09T05:01:13.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.417+0000 7f8ab5a14f40 1 -- 172.21.15.60:0/805020121 >> 172.21.15.60:0/805020121 conn(0x5564a20598d0 msgr2=0x5564a2107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:13.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.417+0000 7f8ab5a14f40 1 -- 172.21.15.60:0/805020121 shutdown_connections 2024-05-09T05:01:13.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.417+0000 7f8ab5a14f40 1 -- 172.21.15.60:0/805020121 wait complete. 2024-05-09T05:01:13.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T05:01:13.422 ERROR:tasks.cephfs.filesystem:Command failed on smithi060 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f6.00000000 symlink' 2024-05-09T05:01:13.422 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_symlink_recovery true 2024-05-09T05:01:13.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.529+0000 7fd3c9589640 1 Processor -- start 2024-05-09T05:01:13.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.529+0000 7fd3c9589640 1 -- start start 2024-05-09T05:01:13.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.530+0000 7fd3c9589640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3c41486d0 0x7fd3c40b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:13.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.530+0000 7fd3c9589640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3c40b6400 con 0x7fd3c40b7080 2024-05-09T05:01:13.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.530+0000 7fd3c9589640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3c40d33a0 con 0x7fd3c4148c40 2024-05-09T05:01:13.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.530+0000 7fd3c9589640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd3c4071810 con 0x7fd3c41486d0 2024-05-09T05:01:13.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.530+0000 7fd3c37fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd3c40b7080 0x7fd3c4072820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33560/0 (socket says 172.21.15.60:33560) 2024-05-09T05:01:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.530+0000 7fd3c37fe640 1 -- 172.21.15.60:0/314889185 learned_addr learned my addr 172.21.15.60:0/314889185 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.530+0000 7fd3c2ffd640 1 --2- 172.21.15.60:0/314889185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3c41486d0 0x7fd3c40b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.530+0000 7fd3c1ffb640 1 -- 172.21.15.60:0/314889185 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 470385353 0 0) 0x7fd3bc001490 con 0x7fd3c40b7080 2024-05-09T05:01:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.530+0000 7fd3c1ffb640 1 -- 172.21.15.60:0/314889185 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3b0003520 con 0x7fd3c40b7080 2024-05-09T05:01:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.531+0000 7fd3c1ffb640 1 -- 172.21.15.60:0/314889185 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3220282600 0 0) 0x7fd3ac001110 con 0x7fd3c4148c40 2024-05-09T05:01:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.531+0000 7fd3c1ffb640 1 -- 172.21.15.60:0/314889185 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3bc001490 con 0x7fd3c4148c40 2024-05-09T05:01:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.531+0000 7fd3c1ffb640 1 -- 172.21.15.60:0/314889185 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1611013168 0 0) 0x7fd3b0003520 con 0x7fd3c40b7080 2024-05-09T05:01:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.531+0000 7fd3c2ffd640 1 -- 172.21.15.60:0/314889185 >> v1:172.21.15.114:6789/0 conn(0x7fd3c4148c40 legacy=0x7fd3c40d2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:13.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.531+0000 7fd3c2ffd640 1 -- 172.21.15.60:0/314889185 >> v1:172.21.15.60:6789/0 conn(0x7fd3c40b7080 legacy=0x7fd3c4072820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:13.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.531+0000 7fd3c2ffd640 1 -- 172.21.15.60:0/314889185 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3c4071a60 con 0x7fd3c41486d0 2024-05-09T05:01:13.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.531+0000 7fd3c2ffd640 1 --2- 172.21.15.60:0/314889185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3c41486d0 0x7fd3c40b5ec0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40097d0 tx=0x7fd3b402a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aef8e106ef1a61b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:13.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.532+0000 7fd3c1ffb640 1 -- 172.21.15.60:0/314889185 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4037070 con 0x7fd3c41486d0 2024-05-09T05:01:13.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.532+0000 7fd3c1ffb640 1 -- 172.21.15.60:0/314889185 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3b402acc0 con 0x7fd3c41486d0 2024-05-09T05:01:13.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.532+0000 7fd3c1ffb640 1 -- 172.21.15.60:0/314889185 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc001c40 con 0x7fd3c41486d0 2024-05-09T05:01:13.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.532+0000 7fd3c9589640 1 -- 172.21.15.60:0/314889185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3c41486d0 msgr2=0x7fd3c40b5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:13.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.532+0000 7fd3c9589640 1 --2- 172.21.15.60:0/314889185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3c41486d0 0x7fd3c40b5ec0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40097d0 tx=0x7fd3b402a230 comp rx=0 tx=0).stop 2024-05-09T05:01:13.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.532+0000 7fd3c9589640 1 -- 172.21.15.60:0/314889185 shutdown_connections 2024-05-09T05:01:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.532+0000 7fd3c9589640 1 --2- 172.21.15.60:0/314889185 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3c41486d0 0x7fd3c40b5ec0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.532+0000 7fd3c9589640 1 -- 172.21.15.60:0/314889185 >> 172.21.15.60:0/314889185 conn(0x7fd3c406f6f0 msgr2=0x7fd3c40ba5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.532+0000 7fd3c9589640 1 -- 172.21.15.60:0/314889185 shutdown_connections 2024-05-09T05:01:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.532+0000 7fd3c9589640 1 -- 172.21.15.60:0/314889185 wait complete. 2024-05-09T05:01:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.534+0000 7fd3c9589640 1 Processor -- start 2024-05-09T05:01:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.534+0000 7fd3c9589640 1 -- start start 2024-05-09T05:01:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.534+0000 7fd3c9589640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3c40b7080 0x7fd3c4162380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.534+0000 7fd3c9589640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3c41486d0 0x7fd3c4136e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.534+0000 7fd3c9589640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3c4148c40 0x7fd3c40b9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.534+0000 7fd3c9589640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c4072300 con 0x7fd3c40b7080 2024-05-09T05:01:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.534+0000 7fd3c9589640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c4072020 con 0x7fd3c41486d0 2024-05-09T05:01:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.534+0000 7fd3c9589640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd3c4072f30 con 0x7fd3c4148c40 2024-05-09T05:01:13.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.534+0000 7fd3c2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3c40b7080 0x7fd3c4162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.534+0000 7fd3c2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3c40b7080 0x7fd3c4162380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59948/0 (socket says 172.21.15.60:59948) 2024-05-09T05:01:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.534+0000 7fd3c2ffd640 1 -- 172.21.15.60:0/2716109908 learned_addr learned my addr 172.21.15.60:0/2716109908 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.535+0000 7fd3c37fe640 1 --2- 172.21.15.60:0/2716109908 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3c4148c40 0x7fd3c40b9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.535+0000 7fd3c27fc640 1 --2- 172.21.15.60:0/2716109908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3c41486d0 0x7fd3c4136e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.535+0000 7fd3c2ffd640 1 -- 172.21.15.60:0/2716109908 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3c4148c40 msgr2=0x7fd3c40b9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.535+0000 7fd3c2ffd640 1 --2- 172.21.15.60:0/2716109908 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3c4148c40 0x7fd3c40b9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.535+0000 7fd3c2ffd640 1 -- 172.21.15.60:0/2716109908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3c41486d0 msgr2=0x7fd3c4136e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.535+0000 7fd3c2ffd640 1 --2- 172.21.15.60:0/2716109908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3c41486d0 0x7fd3c4136e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.535+0000 7fd3c2ffd640 1 -- 172.21.15.60:0/2716109908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3c4137560 con 0x7fd3c40b7080 2024-05-09T05:01:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.535+0000 7fd3c2ffd640 1 --2- 172.21.15.60:0/2716109908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3c40b7080 0x7fd3c4162380 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4035040 tx=0x7fd3b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.535+0000 7fd3bb7fe640 1 -- 172.21.15.60:0/2716109908 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4041020 con 0x7fd3c40b7080 2024-05-09T05:01:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.535+0000 7fd3bb7fe640 1 -- 172.21.15.60:0/2716109908 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3b403d640 con 0x7fd3c40b7080 2024-05-09T05:01:13.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.535+0000 7fd3c9589640 1 -- 172.21.15.60:0/2716109908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3c4137840 con 0x7fd3c40b7080 2024-05-09T05:01:13.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.535+0000 7fd3bb7fe640 1 -- 172.21.15.60:0/2716109908 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4037040 con 0x7fd3c40b7080 2024-05-09T05:01:13.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.535+0000 7fd3c9589640 1 -- 172.21.15.60:0/2716109908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3c4137e00 con 0x7fd3c40b7080 2024-05-09T05:01:13.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.536+0000 7fd3c9589640 1 -- 172.21.15.60:0/2716109908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3c4051040 con 0x7fd3c40b7080 2024-05-09T05:01:13.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.537+0000 7fd3bb7fe640 1 -- 172.21.15.60:0/2716109908 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd3b4044020 con 0x7fd3c40b7080 2024-05-09T05:01:13.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.538+0000 7fd3bb7fe640 1 --2- 172.21.15.60:0/2716109908 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd3a40762d0 0x7fd3a4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:13.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.538+0000 7fd3bb7fe640 1 -- 172.21.15.60:0/2716109908 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7fd3b40b7b40 con 0x7fd3c40b7080 2024-05-09T05:01:13.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.538+0000 7fd3c27fc640 1 --2- 172.21.15.60:0/2716109908 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd3a40762d0 0x7fd3a4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:13.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.538+0000 7fd3c27fc640 1 --2- 172.21.15.60:0/2716109908 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd3a40762d0 0x7fd3a4078790 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd3c40d1cd0 tx=0x7fd3ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:13.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.542+0000 7fd3bb7fe640 1 -- 172.21.15.60:0/2716109908 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd3b404b030 con 0x7fd3c40b7080 2024-05-09T05:01:13.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.750+0000 7fd3c9589640 1 -- 172.21.15.60:0/2716109908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mds_symlink_recovery}] v 0) -- 0x7fd3c4073980 con 0x7fd3c40b7080 2024-05-09T05:01:13.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.753+0000 7fd3bb7fe640 1 -- 172.21.15.60:0/2716109908 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_symlink_recovery}]=0 v12)=0 v12) ==== 121+0+0 (secure 0 0 0) 0x7fd3b4089300 con 0x7fd3c40b7080 2024-05-09T05:01:13.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.755+0000 7fd3c9589640 1 -- 172.21.15.60:0/2716109908 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd3a40762d0 msgr2=0x7fd3a4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:13.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.755+0000 7fd3c9589640 1 --2- 172.21.15.60:0/2716109908 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd3a40762d0 0x7fd3a4078790 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd3c40d1cd0 tx=0x7fd3ac009040 comp rx=0 tx=0).stop 2024-05-09T05:01:13.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.755+0000 7fd3c9589640 1 -- 172.21.15.60:0/2716109908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3c40b7080 msgr2=0x7fd3c4162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:13.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.755+0000 7fd3c9589640 1 --2- 172.21.15.60:0/2716109908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3c40b7080 0x7fd3c4162380 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4035040 tx=0x7fd3b40046c0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.755+0000 7fd3c9589640 1 -- 172.21.15.60:0/2716109908 shutdown_connections 2024-05-09T05:01:13.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.755+0000 7fd3c9589640 1 --2- 172.21.15.60:0/2716109908 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd3a40762d0 0x7fd3a4078790 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.755+0000 7fd3c9589640 1 --2- 172.21.15.60:0/2716109908 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd3c4148c40 0x7fd3c40b9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.755+0000 7fd3c9589640 1 --2- 172.21.15.60:0/2716109908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3c41486d0 0x7fd3c4136e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.755+0000 7fd3c9589640 1 --2- 172.21.15.60:0/2716109908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3c40b7080 0x7fd3c4162380 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:13.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.755+0000 7fd3c9589640 1 -- 172.21.15.60:0/2716109908 >> 172.21.15.60:0/2716109908 conn(0x7fd3c406f6f0 msgr2=0x7fd3c40b8380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:13.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.755+0000 7fd3c9589640 1 -- 172.21.15.60:0/2716109908 shutdown_connections 2024-05-09T05:01:13.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:13.755+0000 7fd3c9589640 1 -- 172.21.15.60:0/2716109908 wait complete. 2024-05-09T05:01:13.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:13.767 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir sym_dir) 2024-05-09T05:01:13.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:13.784 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch sym_dir/file1) 2024-05-09T05:01:13.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:13.843 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln -s ./file1 sym_dir/symlink_file1) 2024-05-09T05:01:13.900 DEBUG:teuthology.orchestra.run.smithi060:> stdin-killer -- python3 -c ' 2024-05-09T05:01:13.901 DEBUG:teuthology.orchestra.run.smithi060:> import os 2024-05-09T05:01:13.901 DEBUG:teuthology.orchestra.run.smithi060:> import stat 2024-05-09T05:01:13.901 DEBUG:teuthology.orchestra.run.smithi060:> 2024-05-09T05:01:13.901 DEBUG:teuthology.orchestra.run.smithi060:> print(os.lstat("/home/ubuntu/cephtest/mnt.0/sym_dir/symlink_file1").st_ino) 2024-05-09T05:01:13.901 DEBUG:teuthology.orchestra.run.smithi060:> ' 2024-05-09T05:01:14.008 INFO:teuthology.orchestra.run.smithi060.stdout:1099511628281 2024-05-09T05:01:14.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:01:14.017 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:14.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.161+0000 7fd25cbda640 1 Processor -- start 2024-05-09T05:01:14.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.161+0000 7fd25cbda640 1 -- start start 2024-05-09T05:01:14.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.162+0000 7fd25cbda640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd258070860 0x7fd258072c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:14.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.162+0000 7fd25cbda640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd2580ac430 0x7fd258073160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:14.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.162+0000 7fd25cbda640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd258071b80 con 0x7fd25815c3a0 2024-05-09T05:01:14.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.162+0000 7fd25cbda640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fd258071e60 con 0x7fd258070860 2024-05-09T05:01:14.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.162+0000 7fd25cbda640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd258072140 con 0x7fd2580ac430 2024-05-09T05:01:14.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.162+0000 7fd256d9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd2580ac430 0x7fd258073160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:14.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.162+0000 7fd255d9b640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd258070860 0x7fd258072c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:14.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.162+0000 7fd256d9d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd2580ac430 0x7fd258073160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.60:60060/0 (socket says 172.21.15.60:60060) 2024-05-09T05:01:14.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.162+0000 7fd256d9d640 1 -- 172.21.15.60:0/4029586509 learned_addr learned my addr 172.21.15.60:0/4029586509 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:14.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.163+0000 7fd255d9b640 1 -- 172.21.15.60:0/4029586509 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd2580ac430 msgr2=0x7fd258073160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:14.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.163+0000 7fd255d9b640 1 --2- 172.21.15.60:0/4029586509 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd2580ac430 0x7fd258073160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:14.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.163+0000 7fd255d9b640 1 -- 172.21.15.60:0/4029586509 >> v1:172.21.15.114:6789/0 conn(0x7fd25815c3a0 legacy=0x7fd258070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:14.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.163+0000 7fd255d9b640 1 -- 172.21.15.60:0/4029586509 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd258072420 con 0x7fd258070860 2024-05-09T05:01:14.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.163+0000 7fd255d9b640 1 --2- 172.21.15.60:0/4029586509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd258070860 0x7fd258072c20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd24c00ba40 tx=0x7fd24c02c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79e3f793e398120e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:14.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.164+0000 7fd25559a640 1 -- 172.21.15.60:0/4029586509 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24c032750 con 0x7fd258070860 2024-05-09T05:01:14.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.164+0000 7fd25559a640 1 -- 172.21.15.60:0/4029586509 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd24c030050 con 0x7fd258070860 2024-05-09T05:01:14.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.164+0000 7fd25559a640 1 -- 172.21.15.60:0/4029586509 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24c032b10 con 0x7fd258070860 2024-05-09T05:01:14.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.164+0000 7fd25cbda640 1 -- 172.21.15.60:0/4029586509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd258070860 msgr2=0x7fd258072c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:14.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.164+0000 7fd25cbda640 1 --2- 172.21.15.60:0/4029586509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd258070860 0x7fd258072c20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd24c00ba40 tx=0x7fd24c02c850 comp rx=0 tx=0).stop 2024-05-09T05:01:14.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.164+0000 7fd25cbda640 1 -- 172.21.15.60:0/4029586509 shutdown_connections 2024-05-09T05:01:14.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.164+0000 7fd25cbda640 1 --2- 172.21.15.60:0/4029586509 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd2580ac430 0x7fd258073160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:14.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.164+0000 7fd25cbda640 1 --2- 172.21.15.60:0/4029586509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd258070860 0x7fd258072c20 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:14.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.164+0000 7fd25cbda640 1 -- 172.21.15.60:0/4029586509 >> 172.21.15.60:0/4029586509 conn(0x7fd2580972f0 msgr2=0x7fd2580976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:14.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.164+0000 7fd25cbda640 1 -- 172.21.15.60:0/4029586509 shutdown_connections 2024-05-09T05:01:14.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.164+0000 7fd25cbda640 1 -- 172.21.15.60:0/4029586509 wait complete. 2024-05-09T05:01:14.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.166+0000 7fd25cbda640 1 Processor -- start 2024-05-09T05:01:14.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.166+0000 7fd25cbda640 1 -- start start 2024-05-09T05:01:14.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.166+0000 7fd25cbda640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd258070860 0x7fd258156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:14.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.166+0000 7fd25cbda640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2580ac430 0x7fd258158180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:14.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.166+0000 7fd25cbda640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd25815c3a0 0x7fd2580b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:14.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.166+0000 7fd25cbda640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd258071810 con 0x7fd258070860 2024-05-09T05:01:14.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.166+0000 7fd25cbda640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd25815c780 con 0x7fd2580ac430 2024-05-09T05:01:14.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.166+0000 7fd25cbda640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd258097de0 con 0x7fd25815c3a0 2024-05-09T05:01:14.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.167+0000 7fd25659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd258070860 0x7fd258156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:14.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.167+0000 7fd25659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd258070860 0x7fd258156010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59964/0 (socket says 172.21.15.60:59964) 2024-05-09T05:01:14.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.167+0000 7fd255d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2580ac430 0x7fd258158180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:14.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.167+0000 7fd25659c640 1 -- 172.21.15.60:0/2435344427 learned_addr learned my addr 172.21.15.60:0/2435344427 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:14.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.167+0000 7fd256d9d640 1 --2- 172.21.15.60:0/2435344427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd25815c3a0 0x7fd2580b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:14.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.167+0000 7fd25659c640 1 -- 172.21.15.60:0/2435344427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd25815c3a0 msgr2=0x7fd2580b0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:14.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.167+0000 7fd25659c640 1 --2- 172.21.15.60:0/2435344427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd25815c3a0 0x7fd2580b0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:14.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.167+0000 7fd25659c640 1 -- 172.21.15.60:0/2435344427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2580ac430 msgr2=0x7fd258158180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:14.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.167+0000 7fd25659c640 1 --2- 172.21.15.60:0/2435344427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2580ac430 0x7fd258158180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:14.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.167+0000 7fd25659c640 1 -- 172.21.15.60:0/2435344427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd258156550 con 0x7fd258070860 2024-05-09T05:01:14.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.167+0000 7fd255d9b640 1 --2- 172.21.15.60:0/2435344427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2580ac430 0x7fd258158180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:01:14.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.167+0000 7fd25659c640 1 --2- 172.21.15.60:0/2435344427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd258070860 0x7fd258156010 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd24800c3a0 tx=0x7fd24800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:14.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.167+0000 7fd246ffd640 1 -- 172.21.15.60:0/2435344427 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd248017070 con 0x7fd258070860 2024-05-09T05:01:14.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.167+0000 7fd246ffd640 1 -- 172.21.15.60:0/2435344427 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2480052d0 con 0x7fd258070860 2024-05-09T05:01:14.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.167+0000 7fd25cbda640 1 -- 172.21.15.60:0/2435344427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2580ac990 con 0x7fd258070860 2024-05-09T05:01:14.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.167+0000 7fd246ffd640 1 -- 172.21.15.60:0/2435344427 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd248003830 con 0x7fd258070860 2024-05-09T05:01:14.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.167+0000 7fd25cbda640 1 -- 172.21.15.60:0/2435344427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2580acf50 con 0x7fd258070860 2024-05-09T05:01:14.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.169+0000 7fd25cbda640 1 -- 172.21.15.60:0/2435344427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd258051040 con 0x7fd258070860 2024-05-09T05:01:14.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.169+0000 7fd246ffd640 1 -- 172.21.15.60:0/2435344427 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd24801a070 con 0x7fd258070860 2024-05-09T05:01:14.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.169+0000 7fd246ffd640 1 --2- 172.21.15.60:0/2435344427 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd2380762d0 0x7fd238078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:14.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.170+0000 7fd246ffd640 1 -- 172.21.15.60:0/2435344427 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7fd248096a40 con 0x7fd258070860 2024-05-09T05:01:14.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.170+0000 7fd255d9b640 1 --2- 172.21.15.60:0/2435344427 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd2380762d0 0x7fd238078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:14.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.170+0000 7fd255d9b640 1 --2- 172.21.15.60:0/2435344427 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd2380762d0 0x7fd238078790 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd24c037000 tx=0x7fd24c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:14.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.174+0000 7fd246ffd640 1 -- 172.21.15.60:0/2435344427 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd24805fbc0 con 0x7fd258070860 2024-05-09T05:01:14.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.404+0000 7fd25cbda640 1 -- 172.21.15.60:0/2435344427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd25809b3f0 con 0x7fd258070860 2024-05-09T05:01:14.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.405+0000 7fd246ffd640 1 -- 172.21.15.60:0/2435344427 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5181 (secure 0 0 0) 0x7fd248068020 con 0x7fd258070860 2024-05-09T05:01:14.406 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:14.406 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4547,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2372252858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2372252858},{"type":"v1","addr":"172.21.15.114:6836","nonce":2372252858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1761282610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1761282610},{"type":"v1","addr":"172.21.15.114:6837","nonce":1761282610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4765,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/300408012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":300408012},{"type":"v1","addr":"172.21.15.60:6835","nonce":300408012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:04.698975+0000","modified":"2024-05-09T05:01:07.722167+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4893},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4893":{"gid":4893,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/399160914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":399160914},{"type":"v1","addr":"172.21.15.60:6834","nonce":399160914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4893,"qdb_cluster":[4893]},"id":2}]} 2024-05-09T05:01:14.406 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 17 2024-05-09T05:01:14.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.407+0000 7fd25cbda640 1 -- 172.21.15.60:0/2435344427 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd2380762d0 msgr2=0x7fd238078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:14.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.407+0000 7fd25cbda640 1 --2- 172.21.15.60:0/2435344427 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd2380762d0 0x7fd238078790 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd24c037000 tx=0x7fd24c032000 comp rx=0 tx=0).stop 2024-05-09T05:01:14.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.407+0000 7fd25cbda640 1 -- 172.21.15.60:0/2435344427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd258070860 msgr2=0x7fd258156010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:14.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.407+0000 7fd25cbda640 1 --2- 172.21.15.60:0/2435344427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd258070860 0x7fd258156010 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd24800c3a0 tx=0x7fd24800c860 comp rx=0 tx=0).stop 2024-05-09T05:01:14.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.407+0000 7fd25cbda640 1 -- 172.21.15.60:0/2435344427 shutdown_connections 2024-05-09T05:01:14.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.407+0000 7fd25cbda640 1 --2- 172.21.15.60:0/2435344427 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd2380762d0 0x7fd238078790 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:14.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.407+0000 7fd25cbda640 1 --2- 172.21.15.60:0/2435344427 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd25815c3a0 0x7fd2580b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:14.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.407+0000 7fd25cbda640 1 --2- 172.21.15.60:0/2435344427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2580ac430 0x7fd258158180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:14.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.407+0000 7fd25cbda640 1 --2- 172.21.15.60:0/2435344427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd258070860 0x7fd258156010 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:14.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.407+0000 7fd25cbda640 1 -- 172.21.15.60:0/2435344427 >> 172.21.15.60:0/2435344427 conn(0x7fd2580972f0 msgr2=0x7fd2580976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:14.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.407+0000 7fd25cbda640 1 -- 172.21.15.60:0/2435344427 shutdown_connections 2024-05-09T05:01:14.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.407+0000 7fd25cbda640 1 -- 172.21.15.60:0/2435344427 wait complete. 2024-05-09T05:01:14.418 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-05-09T05:01:14.554 INFO:teuthology.orchestra.run.smithi060.stdout:{"message":"","return_code":0} 2024-05-09T05:01:14.554 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-05-09T05:01:14.554 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f9.00000000 symlink 2024-05-09T05:01:14.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.599+0000 7fb973121f40 1 Processor -- start 2024-05-09T05:01:14.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.599+0000 7fb973121f40 1 -- start start 2024-05-09T05:01:14.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.599+0000 7fb973121f40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55eb23000fc0 0x55eb2302d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:14.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.599+0000 7fb973121f40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eb2304c340 0x55eb230d8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:14.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.599+0000 7fb973121f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb230c3510 0x55eb230d94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:14.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.599+0000 7fb973121f40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55eb22fa9f50 con 0x55eb2304c340 2024-05-09T05:01:14.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.599+0000 7fb973121f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55eb2304c7b0 con 0x55eb230c3510 2024-05-09T05:01:14.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.599+0000 7fb973121f40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x55eb230c29a0 con 0x55eb23000fc0 2024-05-09T05:01:14.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.599+0000 7fb96f9d8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eb2304c340 0x55eb230d8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:14.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.599+0000 7fb96f9d8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eb2304c340 0x55eb230d8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59966/0 (socket says 172.21.15.60:59966) 2024-05-09T05:01:14.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.599+0000 7fb96f9d8640 1 -- 172.21.15.60:0/558951456 learned_addr learned my addr 172.21.15.60:0/558951456 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:14.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.600+0000 7fb9701d9640 1 --2- 172.21.15.60:0/558951456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55eb23000fc0 0x55eb2302d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:14.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.600+0000 7fb9709da640 1 --2- 172.21.15.60:0/558951456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb230c3510 0x55eb230d94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:14.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.600+0000 7fb96f9d8640 1 -- 172.21.15.60:0/558951456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55eb23000fc0 msgr2=0x55eb2302d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:14.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.600+0000 7fb96f9d8640 1 --2- 172.21.15.60:0/558951456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55eb23000fc0 0x55eb2302d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:14.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.600+0000 7fb96f9d8640 1 -- 172.21.15.60:0/558951456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb230c3510 msgr2=0x55eb230d94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:14.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.600+0000 7fb96f9d8640 1 --2- 172.21.15.60:0/558951456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb230c3510 0x55eb230d94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:14.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.600+0000 7fb96f9d8640 1 -- 172.21.15.60:0/558951456 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb23049130 con 0x55eb2304c340 2024-05-09T05:01:14.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.600+0000 7fb96f9d8640 1 --2- 172.21.15.60:0/558951456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eb2304c340 0x55eb230d8f90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb95800b7c0 tx=0x7fb95800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1e89bbb752d811 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:14.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.600+0000 7fb96e9d6640 1 -- 172.21.15.60:0/558951456 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb95800abb0 con 0x55eb2304c340 2024-05-09T05:01:14.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.600+0000 7fb96e9d6640 1 -- 172.21.15.60:0/558951456 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb95800ad50 con 0x55eb2304c340 2024-05-09T05:01:14.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.600+0000 7fb96e9d6640 1 -- 172.21.15.60:0/558951456 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb958015670 con 0x55eb2304c340 2024-05-09T05:01:14.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.601+0000 7fb973121f40 1 -- 172.21.15.60:0/558951456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eb2304c340 msgr2=0x55eb230d8f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:14.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.601+0000 7fb973121f40 1 --2- 172.21.15.60:0/558951456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eb2304c340 0x55eb230d8f90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb95800b7c0 tx=0x7fb95800bc80 comp rx=0 tx=0).stop 2024-05-09T05:01:14.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.601+0000 7fb973121f40 1 -- 172.21.15.60:0/558951456 shutdown_connections 2024-05-09T05:01:14.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.601+0000 7fb973121f40 1 --2- 172.21.15.60:0/558951456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb230c3510 0x55eb230d94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:14.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.601+0000 7fb973121f40 1 --2- 172.21.15.60:0/558951456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eb2304c340 0x55eb230d8f90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:14.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.601+0000 7fb973121f40 1 --2- 172.21.15.60:0/558951456 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55eb23000fc0 0x55eb2302d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:14.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.601+0000 7fb973121f40 1 -- 172.21.15.60:0/558951456 >> 172.21.15.60:0/558951456 conn(0x55eb2302c8d0 msgr2=0x55eb230046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:14.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.601+0000 7fb973121f40 1 -- 172.21.15.60:0/558951456 shutdown_connections 2024-05-09T05:01:14.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.601+0000 7fb973121f40 1 -- 172.21.15.60:0/558951456 wait complete. 2024-05-09T05:01:14.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.601+0000 7fb973121f40 1 Processor -- start 2024-05-09T05:01:14.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.601+0000 7fb973121f40 1 -- start start 2024-05-09T05:01:14.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.602+0000 7fb973121f40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55eb23000fc0 0x55eb230dc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:14.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.602+0000 7fb973121f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb2304c340 0x55eb23045fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:14.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.602+0000 7fb973121f40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eb230c3510 0x55eb23046520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:14.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.602+0000 7fb973121f40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55eb23031310 con 0x55eb230c3510 2024-05-09T05:01:14.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.602+0000 7fb973121f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55eb2304c0c0 con 0x55eb2304c340 2024-05-09T05:01:14.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.602+0000 7fb973121f40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x55eb230c2bf0 con 0x55eb23000fc0 2024-05-09T05:01:14.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.602+0000 7fb9709da640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eb230c3510 0x55eb23046520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:14.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.602+0000 7fb9709da640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eb230c3510 0x55eb23046520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59972/0 (socket says 172.21.15.60:59972) 2024-05-09T05:01:14.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.602+0000 7fb9709da640 1 -- 172.21.15.60:0/2858856926 learned_addr learned my addr 172.21.15.60:0/2858856926 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:14.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.602+0000 7fb96f9d8640 1 --2- 172.21.15.60:0/2858856926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb2304c340 0x55eb23045fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:14.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.602+0000 7fb9701d9640 1 --2- 172.21.15.60:0/2858856926 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55eb23000fc0 0x55eb230dc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:14.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.602+0000 7fb9709da640 1 -- 172.21.15.60:0/2858856926 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55eb23000fc0 msgr2=0x55eb230dc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:14.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.602+0000 7fb9709da640 1 --2- 172.21.15.60:0/2858856926 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55eb23000fc0 0x55eb230dc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:14.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.602+0000 7fb9709da640 1 -- 172.21.15.60:0/2858856926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb2304c340 msgr2=0x55eb23045fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:14.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.602+0000 7fb9709da640 1 --2- 172.21.15.60:0/2858856926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb2304c340 0x55eb23045fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:14.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.602+0000 7fb9709da640 1 -- 172.21.15.60:0/2858856926 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb230bb610 con 0x55eb230c3510 2024-05-09T05:01:14.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.603+0000 7fb9709da640 1 --2- 172.21.15.60:0/2858856926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eb230c3510 0x55eb23046520 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb96800c4b0 tx=0x7fb96800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:14.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.603+0000 7fb96d9d4640 1 -- 172.21.15.60:0/2858856926 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb968008070 con 0x55eb230c3510 2024-05-09T05:01:14.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.603+0000 7fb96d9d4640 1 -- 172.21.15.60:0/2858856926 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9680116a0 con 0x55eb230c3510 2024-05-09T05:01:14.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.603+0000 7fb973121f40 1 -- 172.21.15.60:0/2858856926 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55eb230bb330 con 0x55eb230c3510 2024-05-09T05:01:14.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.603+0000 7fb96d9d4640 1 -- 172.21.15.60:0/2858856926 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb968011a90 con 0x55eb230c3510 2024-05-09T05:01:14.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.603+0000 7fb973121f40 1 -- 172.21.15.60:0/2858856926 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb230bc190 con 0x55eb230c3510 2024-05-09T05:01:14.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.604+0000 7fb96d9d4640 1 -- 172.21.15.60:0/2858856926 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb968011c30 con 0x55eb230c3510 2024-05-09T05:01:14.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.605+0000 7fb96d9d4640 1 --2- 172.21.15.60:0/2858856926 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb950074cb0 0x7fb950077170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:14.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.605+0000 7fb96d9d4640 1 -- 172.21.15.60:0/2858856926 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7fb968096ba0 con 0x55eb230c3510 2024-05-09T05:01:14.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.605+0000 7fb9701d9640 1 --2- 172.21.15.60:0/2858856926 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb950074cb0 0x7fb950077170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:14.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.605+0000 7fb973121f40 1 --2- 172.21.15.60:0/2858856926 >> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] conn(0x55eb230bacb0 0x55eb230475d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:14.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.605+0000 7fb973121f40 1 -- 172.21.15.60:0/2858856926 --> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] -- osd_op(unknown.0.0:1 6.36 6:6ee5e2c4:::100000001f9.00000000:head [getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) -- 0x55eb230be5d0 con 0x55eb230bacb0 2024-05-09T05:01:14.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.605+0000 7fb96f9d8640 1 --2- 172.21.15.60:0/2858856926 >> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] conn(0x55eb230bacb0 0x55eb230475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:14.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.605+0000 7fb9701d9640 1 --2- 172.21.15.60:0/2858856926 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb950074cb0 0x7fb950077170 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb960056000 tx=0x7fb960007030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:14.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.606+0000 7fb96f9d8640 1 --2- 172.21.15.60:0/2858856926 >> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] conn(0x55eb230bacb0 0x55eb230475d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:14.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.606+0000 7fb96f9d8640 1 -- 172.21.15.60:0/2858856926 <== osd.5 v2:172.21.15.114:6816/292113074 1 ==== osd_op_reply(1 100000001f9.00000000 [getxattr (11) out=11b] v0'0 uv1 ondisk = 0) ==== 164+0+11 (crc 0 0 0) 0x7fb958015470 con 0x55eb230bacb0 2024-05-09T05:01:14.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.606+0000 7fb973121f40 1 -- 172.21.15.60:0/2858856926 >> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] conn(0x55eb230bacb0 msgr2=0x55eb230475d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:14.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.606+0000 7fb973121f40 1 --2- 172.21.15.60:0/2858856926 >> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] conn(0x55eb230bacb0 0x55eb230475d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:14.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.606+0000 7fb973121f40 1 -- 172.21.15.60:0/2858856926 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb950074cb0 msgr2=0x7fb950077170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:14.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.606+0000 7fb973121f40 1 --2- 172.21.15.60:0/2858856926 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb950074cb0 0x7fb950077170 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb960056000 tx=0x7fb960007030 comp rx=0 tx=0).stop 2024-05-09T05:01:14.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.606+0000 7fb973121f40 1 -- 172.21.15.60:0/2858856926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eb230c3510 msgr2=0x55eb23046520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:14.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.606+0000 7fb973121f40 1 --2- 172.21.15.60:0/2858856926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eb230c3510 0x55eb23046520 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb96800c4b0 tx=0x7fb96800c8b0 comp rx=0 tx=0).stop 2024-05-09T05:01:14.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.606+0000 7fb9709da640 1 -- 172.21.15.60:0/2858856926 reap_dead start 2024-05-09T05:01:14.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.606+0000 7fb973121f40 1 -- 172.21.15.60:0/2858856926 shutdown_connections 2024-05-09T05:01:14.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.606+0000 7fb973121f40 1 -- 172.21.15.60:0/2858856926 >> 172.21.15.60:0/2858856926 conn(0x55eb2302c8d0 msgr2=0x55eb230c1b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:14.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.606+0000 7fb973121f40 1 -- 172.21.15.60:0/2858856926 shutdown_connections 2024-05-09T05:01:14.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14.606+0000 7fb973121f40 1 -- 172.21.15.60:0/2858856926 wait complete. 2024-05-09T05:01:14.611 INFO:teuthology.orchestra.run.smithi060.stdout:./file1 2024-05-09T05:01:14.612 DEBUG:teuthology.orchestra.run.smithi060:> ceph-dencoder type string_wrapper import - decode dump_json 2024-05-09T05:01:14.702 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:01:14.702 INFO:teuthology.orchestra.run.smithi060.stdout: "s": "./file1" 2024-05-09T05:01:14.702 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:01:14.702 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:14.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:14.706 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent_a) 2024-05-09T05:01:14.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:14.723 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parent_a/alpha) 2024-05-09T05:01:14.783 DEBUG:teuthology.orchestra.run.smithi060:> stdin-killer -- python3 -c ' 2024-05-09T05:01:14.783 DEBUG:teuthology.orchestra.run.smithi060:> import os 2024-05-09T05:01:14.783 DEBUG:teuthology.orchestra.run.smithi060:> import stat 2024-05-09T05:01:14.783 DEBUG:teuthology.orchestra.run.smithi060:> 2024-05-09T05:01:14.783 DEBUG:teuthology.orchestra.run.smithi060:> print(os.stat("/home/ubuntu/cephtest/mnt.0/parent_a/alpha").st_ino) 2024-05-09T05:01:14.783 DEBUG:teuthology.orchestra.run.smithi060:> ' 2024-05-09T05:01:14.891 INFO:teuthology.orchestra.run.smithi060.stdout:1099511628283 2024-05-09T05:01:14.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:01:14.900 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:15.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.044+0000 7f42c5d72640 1 Processor -- start 2024-05-09T05:01:15.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.044+0000 7f42c5d72640 1 -- start start 2024-05-09T05:01:15.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.045+0000 7f42c5d72640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f42c0096d70 0x7f42c009d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.045+0000 7f42c5d72640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f42c00b1040 0x7f42c00b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.045+0000 7f42c5d72640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42c00aea30 0x7f42c00aee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.045+0000 7f42c5d72640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f42c0097150 con 0x7f42c0096d70 2024-05-09T05:01:15.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.045+0000 7f42c5d72640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f42c009c020 con 0x7f42c00b1040 2024-05-09T05:01:15.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.045+0000 7f42c5d72640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f42c009c300 con 0x7f42c00aea30 2024-05-09T05:01:15.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.045+0000 7f42bf7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f42c0096d70 0x7f42c009d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.045+0000 7f42bf7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f42c0096d70 0x7f42c009d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59986/0 (socket says 172.21.15.60:59986) 2024-05-09T05:01:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.045+0000 7f42bf7fe640 1 -- 172.21.15.60:0/4287067673 learned_addr learned my addr 172.21.15.60:0/4287067673 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.045+0000 7f42beffd640 1 --2- 172.21.15.60:0/4287067673 >> v2:172.21.15.114:3300/0 conn(0x7f42c00b1040 0x7f42c00b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.045+0000 7f42bffff640 1 --2- 172.21.15.60:0/4287067673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42c00aea30 0x7f42c00aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.045+0000 7f42bf7fe640 1 -- 172.21.15.60:0/4287067673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42c00aea30 msgr2=0x7f42c00aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.045+0000 7f42bf7fe640 1 --2- 172.21.15.60:0/4287067673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42c00aea30 0x7f42c00aee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.045+0000 7f42bf7fe640 1 -- 172.21.15.60:0/4287067673 >> v2:172.21.15.114:3300/0 conn(0x7f42c00b1040 msgr2=0x7f42c00b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.045+0000 7f42bf7fe640 1 --2- 172.21.15.60:0/4287067673 >> v2:172.21.15.114:3300/0 conn(0x7f42c00b1040 0x7f42c00b14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.045+0000 7f42bf7fe640 1 -- 172.21.15.60:0/4287067673 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c009c5e0 con 0x7f42c0096d70 2024-05-09T05:01:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.045+0000 7f42beffd640 1 --2- 172.21.15.60:0/4287067673 >> v2:172.21.15.114:3300/0 conn(0x7f42c00b1040 0x7f42c00b14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.045+0000 7f42bffff640 1 --2- 172.21.15.60:0/4287067673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42c00aea30 0x7f42c00aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.046+0000 7f42bf7fe640 1 --2- 172.21.15.60:0/4287067673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42c0096d70 0x7f42c009d180 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f42c00743f0 tx=0x7f42b002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77ef6e2a3098e50 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.046+0000 7f42be7fc640 1 -- 172.21.15.60:0/4287067673 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b0037070 con 0x7f42c0096d70 2024-05-09T05:01:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.046+0000 7f42be7fc640 1 -- 172.21.15.60:0/4287067673 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42b0032440 con 0x7f42c0096d70 2024-05-09T05:01:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.046+0000 7f42be7fc640 1 -- 172.21.15.60:0/4287067673 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b0032810 con 0x7f42c0096d70 2024-05-09T05:01:15.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.047+0000 7f42c5d72640 1 -- 172.21.15.60:0/4287067673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42c0096d70 msgr2=0x7f42c009d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:15.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.047+0000 7f42c5d72640 1 --2- 172.21.15.60:0/4287067673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42c0096d70 0x7f42c009d180 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f42c00743f0 tx=0x7f42b002a1e0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.047+0000 7f42c5d72640 1 -- 172.21.15.60:0/4287067673 shutdown_connections 2024-05-09T05:01:15.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.047+0000 7f42c5d72640 1 --2- 172.21.15.60:0/4287067673 >> v2:172.21.15.114:3300/0 conn(0x7f42c00b1040 0x7f42c00b14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.047+0000 7f42c5d72640 1 --2- 172.21.15.60:0/4287067673 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42c00aea30 0x7f42c00aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.047+0000 7f42c5d72640 1 --2- 172.21.15.60:0/4287067673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42c0096d70 0x7f42c009d180 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.047+0000 7f42c5d72640 1 -- 172.21.15.60:0/4287067673 >> 172.21.15.60:0/4287067673 conn(0x7f42c0072390 msgr2=0x7f42c00727a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:15.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.047+0000 7f42c5d72640 1 -- 172.21.15.60:0/4287067673 shutdown_connections 2024-05-09T05:01:15.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.047+0000 7f42c5d72640 1 -- 172.21.15.60:0/4287067673 wait complete. 2024-05-09T05:01:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.048+0000 7f42c5d72640 1 Processor -- start 2024-05-09T05:01:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.048+0000 7f42c5d72640 1 -- start start 2024-05-09T05:01:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.049+0000 7f42c5d72640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42c0096d70 0x7f42c0156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.049+0000 7f42c5d72640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42c00aea30 0x7f42c0162670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.049+0000 7f42c5d72640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c00b1040 0x7f42c015b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.049+0000 7f42c5d72640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f42c00b19f0 con 0x7f42c0096d70 2024-05-09T05:01:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.049+0000 7f42c5d72640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f42c009cba0 con 0x7f42c00b1040 2024-05-09T05:01:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.049+0000 7f42c5d72640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f42c009bc40 con 0x7f42c00aea30 2024-05-09T05:01:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.049+0000 7f42bf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42c0096d70 0x7f42c0156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.049+0000 7f42bf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42c0096d70 0x7f42c0156f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59994/0 (socket says 172.21.15.60:59994) 2024-05-09T05:01:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.049+0000 7f42bf7fe640 1 -- 172.21.15.60:0/2281186812 learned_addr learned my addr 172.21.15.60:0/2281186812 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.049+0000 7f42beffd640 1 --2- 172.21.15.60:0/2281186812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42c00aea30 0x7f42c0162670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.049+0000 7f42bffff640 1 --2- 172.21.15.60:0/2281186812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c00b1040 0x7f42c015b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.049+0000 7f42bf7fe640 1 -- 172.21.15.60:0/2281186812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42c00aea30 msgr2=0x7f42c0162670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.049+0000 7f42bf7fe640 1 --2- 172.21.15.60:0/2281186812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42c00aea30 0x7f42c0162670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.049+0000 7f42bf7fe640 1 -- 172.21.15.60:0/2281186812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c00b1040 msgr2=0x7f42c015b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.049+0000 7f42bf7fe640 1 --2- 172.21.15.60:0/2281186812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c00b1040 0x7f42c015b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.049+0000 7f42bf7fe640 1 -- 172.21.15.60:0/2281186812 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c015fcf0 con 0x7f42c0096d70 2024-05-09T05:01:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.049+0000 7f42bf7fe640 1 --2- 172.21.15.60:0/2281186812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42c0096d70 0x7f42c0156f00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f42b002a220 tx=0x7f42b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.050+0000 7f42a7fff640 1 -- 172.21.15.60:0/2281186812 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b0037040 con 0x7f42c0096d70 2024-05-09T05:01:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.050+0000 7f42a7fff640 1 -- 172.21.15.60:0/2281186812 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42b0042070 con 0x7f42c0096d70 2024-05-09T05:01:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.050+0000 7f42c5d72640 1 -- 172.21.15.60:0/2281186812 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42c01575d0 con 0x7f42c0096d70 2024-05-09T05:01:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.050+0000 7f42a7fff640 1 -- 172.21.15.60:0/2281186812 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b0030bb0 con 0x7f42c0096d70 2024-05-09T05:01:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.050+0000 7f42c5d72640 1 -- 172.21.15.60:0/2281186812 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42c0157b90 con 0x7f42c0096d70 2024-05-09T05:01:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.051+0000 7f42c5d72640 1 -- 172.21.15.60:0/2281186812 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42c0051040 con 0x7f42c0096d70 2024-05-09T05:01:15.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.052+0000 7f42a7fff640 1 -- 172.21.15.60:0/2281186812 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f42b0047460 con 0x7f42c0096d70 2024-05-09T05:01:15.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.052+0000 7f42a7fff640 1 --2- 172.21.15.60:0/2281186812 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f42ac0762d0 0x7f42ac078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.052+0000 7f42a7fff640 1 -- 172.21.15.60:0/2281186812 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7f42b00b8420 con 0x7f42c0096d70 2024-05-09T05:01:15.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.052+0000 7f42beffd640 1 --2- 172.21.15.60:0/2281186812 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f42ac0762d0 0x7f42ac078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.053+0000 7f42beffd640 1 --2- 172.21.15.60:0/2281186812 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f42ac0762d0 0x7f42ac078790 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f42c015df90 tx=0x7f42b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:15.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.056+0000 7f42a7fff640 1 -- 172.21.15.60:0/2281186812 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f42b0081bf0 con 0x7f42c0096d70 2024-05-09T05:01:15.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.289+0000 7f42c5d72640 1 -- 172.21.15.60:0/2281186812 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42c00b3830 con 0x7f42c0096d70 2024-05-09T05:01:15.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.289+0000 7f42a7fff640 1 -- 172.21.15.60:0/2281186812 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5181 (secure 0 0 0) 0x7f42b00814f0 con 0x7f42c0096d70 2024-05-09T05:01:15.290 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:15.290 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4547,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2372252858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2372252858},{"type":"v1","addr":"172.21.15.114:6836","nonce":2372252858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1761282610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1761282610},{"type":"v1","addr":"172.21.15.114:6837","nonce":1761282610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4765,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/300408012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":300408012},{"type":"v1","addr":"172.21.15.60:6835","nonce":300408012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:04.698975+0000","modified":"2024-05-09T05:01:07.722167+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4893},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4893":{"gid":4893,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/399160914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":399160914},{"type":"v1","addr":"172.21.15.60:6834","nonce":399160914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4893,"qdb_cluster":[4893]},"id":2}]} 2024-05-09T05:01:15.290 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 17 2024-05-09T05:01:15.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.292+0000 7f42c5d72640 1 -- 172.21.15.60:0/2281186812 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f42ac0762d0 msgr2=0x7f42ac078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:15.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.292+0000 7f42c5d72640 1 --2- 172.21.15.60:0/2281186812 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f42ac0762d0 0x7f42ac078790 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f42c015df90 tx=0x7f42b4008040 comp rx=0 tx=0).stop 2024-05-09T05:01:15.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.292+0000 7f42c5d72640 1 -- 172.21.15.60:0/2281186812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42c0096d70 msgr2=0x7f42c0156f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:15.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.292+0000 7f42c5d72640 1 --2- 172.21.15.60:0/2281186812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42c0096d70 0x7f42c0156f00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f42b002a220 tx=0x7f42b00046c0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.292+0000 7f42c5d72640 1 -- 172.21.15.60:0/2281186812 shutdown_connections 2024-05-09T05:01:15.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.292+0000 7f42c5d72640 1 --2- 172.21.15.60:0/2281186812 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f42ac0762d0 0x7f42ac078790 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.292+0000 7f42c5d72640 1 --2- 172.21.15.60:0/2281186812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c00b1040 0x7f42c015b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.292+0000 7f42c5d72640 1 --2- 172.21.15.60:0/2281186812 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f42c00aea30 0x7f42c0162670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.292+0000 7f42c5d72640 1 --2- 172.21.15.60:0/2281186812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42c0096d70 0x7f42c0156f00 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.292+0000 7f42c5d72640 1 -- 172.21.15.60:0/2281186812 >> 172.21.15.60:0/2281186812 conn(0x7f42c0072390 msgr2=0x7f42c00aee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:15.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.292+0000 7f42c5d72640 1 -- 172.21.15.60:0/2281186812 shutdown_connections 2024-05-09T05:01:15.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.292+0000 7f42c5d72640 1 -- 172.21.15.60:0/2281186812 wait complete. 2024-05-09T05:01:15.304 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-05-09T05:01:15.434 INFO:teuthology.orchestra.run.smithi060.stdout:{"message":"","return_code":0} 2024-05-09T05:01:15.435 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-05-09T05:01:15.435 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001fb.00000000 parent 2024-05-09T05:01:15.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.479+0000 7f0c26193f40 1 Processor -- start 2024-05-09T05:01:15.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.479+0000 7f0c26193f40 1 -- start start 2024-05-09T05:01:15.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.479+0000 7f0c26193f40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556a7211bc40 0x556a721a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.479+0000 7f0c26193f40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a721346b0 0x556a721c0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.479+0000 7f0c26193f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a721aacd0 0x556a721a9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.479+0000 7f0c26193f40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x556a721333b0 con 0x556a721346b0 2024-05-09T05:01:15.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.479+0000 7f0c26193f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556a7211c0b0 con 0x556a721aacd0 2024-05-09T05:01:15.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.479+0000 7f0c26193f40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x556a72133690 con 0x556a7211bc40 2024-05-09T05:01:15.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.480+0000 7f0c2348c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a721346b0 0x556a721c0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.480+0000 7f0c2348c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a721346b0 0x556a721c0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60014/0 (socket says 172.21.15.60:60014) 2024-05-09T05:01:15.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.480+0000 7f0c2348c640 1 -- 172.21.15.60:0/571916520 learned_addr learned my addr 172.21.15.60:0/571916520 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:15.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.480+0000 7f0c2448e640 1 --2- 172.21.15.60:0/571916520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a721aacd0 0x556a721a9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.480+0000 7f0c23c8d640 1 --2- 172.21.15.60:0/571916520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556a7211bc40 0x556a721a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.480+0000 7f0c2348c640 1 -- 172.21.15.60:0/571916520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556a7211bc40 msgr2=0x556a721a4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:15.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.480+0000 7f0c2348c640 1 --2- 172.21.15.60:0/571916520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556a7211bc40 0x556a721a4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.480+0000 7f0c2348c640 1 -- 172.21.15.60:0/571916520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a721aacd0 msgr2=0x556a721a9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:15.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.480+0000 7f0c2348c640 1 --2- 172.21.15.60:0/571916520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a721aacd0 0x556a721a9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.480+0000 7f0c2348c640 1 -- 172.21.15.60:0/571916520 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a72055350 con 0x556a721346b0 2024-05-09T05:01:15.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.480+0000 7f0c2348c640 1 --2- 172.21.15.60:0/571916520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a721346b0 0x556a721c0ce0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0c18025da0 tx=0x7f0c18051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffd68b63bd5c4f78 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:15.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.480+0000 7f0c2248a640 1 -- 172.21.15.60:0/571916520 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c18052b10 con 0x556a721346b0 2024-05-09T05:01:15.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.480+0000 7f0c2248a640 1 -- 172.21.15.60:0/571916520 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c18052cd0 con 0x556a721346b0 2024-05-09T05:01:15.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.480+0000 7f0c2248a640 1 -- 172.21.15.60:0/571916520 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c180676c0 con 0x556a721346b0 2024-05-09T05:01:15.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.481+0000 7f0c26193f40 1 -- 172.21.15.60:0/571916520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a721346b0 msgr2=0x556a721c0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:15.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.481+0000 7f0c26193f40 1 --2- 172.21.15.60:0/571916520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a721346b0 0x556a721c0ce0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0c18025da0 tx=0x7f0c18051000 comp rx=0 tx=0).stop 2024-05-09T05:01:15.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.481+0000 7f0c26193f40 1 -- 172.21.15.60:0/571916520 shutdown_connections 2024-05-09T05:01:15.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.481+0000 7f0c26193f40 1 --2- 172.21.15.60:0/571916520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a721aacd0 0x556a721a9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.481+0000 7f0c26193f40 1 --2- 172.21.15.60:0/571916520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a721346b0 0x556a721c0ce0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.481+0000 7f0c26193f40 1 --2- 172.21.15.60:0/571916520 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556a7211bc40 0x556a721a4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.481+0000 7f0c26193f40 1 -- 172.21.15.60:0/571916520 >> 172.21.15.60:0/571916520 conn(0x556a720eb890 msgr2=0x556a72117480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:15.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.481+0000 7f0c26193f40 1 -- 172.21.15.60:0/571916520 shutdown_connections 2024-05-09T05:01:15.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.481+0000 7f0c26193f40 1 -- 172.21.15.60:0/571916520 wait complete. 2024-05-09T05:01:15.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.481+0000 7f0c26193f40 1 Processor -- start 2024-05-09T05:01:15.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.481+0000 7f0c26193f40 1 -- start start 2024-05-09T05:01:15.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.482+0000 7f0c26193f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a7211bc40 0x556a721a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.482+0000 7f0c26193f40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a721346b0 0x556a721a3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.482+0000 7f0c26193f40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556a721aacd0 0x556a721a6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.482+0000 7f0c26193f40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x556a721333b0 con 0x556a721346b0 2024-05-09T05:01:15.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.482+0000 7f0c26193f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556a72115d50 con 0x556a7211bc40 2024-05-09T05:01:15.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.482+0000 7f0c26193f40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x556a721158c0 con 0x556a721aacd0 2024-05-09T05:01:15.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.482+0000 7f0c2348c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a721346b0 0x556a721a3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.482+0000 7f0c2348c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a721346b0 0x556a721a3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60028/0 (socket says 172.21.15.60:60028) 2024-05-09T05:01:15.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.482+0000 7f0c2348c640 1 -- 172.21.15.60:0/2258301581 learned_addr learned my addr 172.21.15.60:0/2258301581 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:15.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.482+0000 7f0c23c8d640 1 --2- 172.21.15.60:0/2258301581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a7211bc40 0x556a721a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.482+0000 7f0c2448e640 1 --2- 172.21.15.60:0/2258301581 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556a721aacd0 0x556a721a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.482+0000 7f0c2348c640 1 -- 172.21.15.60:0/2258301581 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556a721aacd0 msgr2=0x556a721a6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:15.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.482+0000 7f0c2348c640 1 --2- 172.21.15.60:0/2258301581 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x556a721aacd0 0x556a721a6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.482+0000 7f0c2348c640 1 -- 172.21.15.60:0/2258301581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a7211bc40 msgr2=0x556a721a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:15.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.482+0000 7f0c2348c640 1 --2- 172.21.15.60:0/2258301581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556a7211bc40 0x556a721a0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.483+0000 7f0c2348c640 1 -- 172.21.15.60:0/2258301581 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a721c1fb0 con 0x556a721346b0 2024-05-09T05:01:15.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.483+0000 7f0c2348c640 1 --2- 172.21.15.60:0/2258301581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a721346b0 0x556a721a3c80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0c18045fd0 tx=0x7f0c18002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:15.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.483+0000 7f0c21488640 1 -- 172.21.15.60:0/2258301581 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1806c070 con 0x556a721346b0 2024-05-09T05:01:15.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.483+0000 7f0c21488640 1 -- 172.21.15.60:0/2258301581 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c18067480 con 0x556a721346b0 2024-05-09T05:01:15.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.483+0000 7f0c26193f40 1 -- 172.21.15.60:0/2258301581 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556a721c2290 con 0x556a721346b0 2024-05-09T05:01:15.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.483+0000 7f0c21488640 1 -- 172.21.15.60:0/2258301581 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c18052b30 con 0x556a721346b0 2024-05-09T05:01:15.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.483+0000 7f0c26193f40 1 -- 172.21.15.60:0/2258301581 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a721c2570 con 0x556a721346b0 2024-05-09T05:01:15.490 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.484+0000 7f0c21488640 1 -- 172.21.15.60:0/2258301581 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0c18079020 con 0x556a721346b0 2024-05-09T05:01:15.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.485+0000 7f0c21488640 1 --2- 172.21.15.60:0/2258301581 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0c10074cb0 0x7f0c10077170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.485+0000 7f0c21488640 1 -- 172.21.15.60:0/2258301581 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7f0c180dbb90 con 0x556a721346b0 2024-05-09T05:01:15.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.485+0000 7f0c26193f40 1 --2- 172.21.15.60:0/2258301581 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x556a721c1ad0 0x556a721c4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.485+0000 7f0c26193f40 1 -- 172.21.15.60:0/2258301581 --> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] -- osd_op(unknown.0.0:1 6.13 6:ca8fdbba:::100000001fb.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) -- 0x556a721c4e90 con 0x556a721c1ad0 2024-05-09T05:01:15.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.485+0000 7f0c23c8d640 1 --2- 172.21.15.60:0/2258301581 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0c10074cb0 0x7f0c10077170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.485+0000 7f0c2448e640 1 --2- 172.21.15.60:0/2258301581 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x556a721c1ad0 0x556a721c4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.486+0000 7f0c2448e640 1 --2- 172.21.15.60:0/2258301581 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x556a721c1ad0 0x556a721c4950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:15.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.486+0000 7f0c23c8d640 1 --2- 172.21.15.60:0/2258301581 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0c10074cb0 0x7f0c10077170 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0c14005fd0 tx=0x7f0c14007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:15.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.486+0000 7f0c2448e640 1 -- 172.21.15.60:0/2258301581 <== osd.3 v2:172.21.15.60:6824/1069806621 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (95) out=95b] v0'0 uv1 ondisk = 0) ==== 164+0+95 (crc 0 0 0) 0x7f0c1c00a150 con 0x556a721c1ad0 2024-05-09T05:01:15.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.486+0000 7f0c26193f40 1 -- 172.21.15.60:0/2258301581 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x556a721c1ad0 msgr2=0x556a721c4950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:15.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.486+0000 7f0c26193f40 1 --2- 172.21.15.60:0/2258301581 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x556a721c1ad0 0x556a721c4950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.486+0000 7f0c26193f40 1 -- 172.21.15.60:0/2258301581 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0c10074cb0 msgr2=0x7f0c10077170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:15.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.486+0000 7f0c26193f40 1 --2- 172.21.15.60:0/2258301581 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0c10074cb0 0x7f0c10077170 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0c14005fd0 tx=0x7f0c14007040 comp rx=0 tx=0).stop 2024-05-09T05:01:15.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.486+0000 7f0c26193f40 1 -- 172.21.15.60:0/2258301581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a721346b0 msgr2=0x556a721a3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:15.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.486+0000 7f0c26193f40 1 --2- 172.21.15.60:0/2258301581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556a721346b0 0x556a721a3c80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0c18045fd0 tx=0x7f0c18002fb0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.486+0000 7f0c2448e640 1 -- 172.21.15.60:0/2258301581 reap_dead start 2024-05-09T05:01:15.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.486+0000 7f0c26193f40 1 -- 172.21.15.60:0/2258301581 shutdown_connections 2024-05-09T05:01:15.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.486+0000 7f0c26193f40 1 -- 172.21.15.60:0/2258301581 >> 172.21.15.60:0/2258301581 conn(0x556a720eb890 msgr2=0x556a721c01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:15.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.486+0000 7f0c26193f40 1 -- 172.21.15.60:0/2258301581 shutdown_connections 2024-05-09T05:01:15.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.486+0000 7f0c26193f40 1 -- 172.21.15.60:0/2258301581 wait complete. 2024-05-09T05:01:15.493 INFO:teuthology.orchestra.run.smithi060.stdout:Y��alphaparent_a 2024-05-09T05:01:15.493 DEBUG:teuthology.orchestra.run.smithi060:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-05-09T05:01:15.583 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:01:15.583 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628283, 2024-05-09T05:01:15.583 INFO:teuthology.orchestra.run.smithi060.stdout: "ancestors": [ 2024-05-09T05:01:15.583 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:15.583 INFO:teuthology.orchestra.run.smithi060.stdout: "dirino": 1099511628282, 2024-05-09T05:01:15.583 INFO:teuthology.orchestra.run.smithi060.stdout: "dname": "alpha", 2024-05-09T05:01:15.583 INFO:teuthology.orchestra.run.smithi060.stdout: "version": 2 2024-05-09T05:01:15.583 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:15.583 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:15.583 INFO:teuthology.orchestra.run.smithi060.stdout: "dirino": 1, 2024-05-09T05:01:15.583 INFO:teuthology.orchestra.run.smithi060.stdout: "dname": "parent_a", 2024-05-09T05:01:15.584 INFO:teuthology.orchestra.run.smithi060.stdout: "version": 18 2024-05-09T05:01:15.584 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:15.584 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-05-09T05:01:15.584 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 6, 2024-05-09T05:01:15.584 INFO:teuthology.orchestra.run.smithi060.stdout: "old_pools": [] 2024-05-09T05:01:15.584 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:01:15.584 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:15.585 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001fb.00000000 layout 2024-05-09T05:01:15.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.670+0000 7f7626cb1f40 1 Processor -- start 2024-05-09T05:01:15.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.670+0000 7f7626cb1f40 1 -- start start 2024-05-09T05:01:15.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.671+0000 7f7626cb1f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7610003ab0 0x5578cd454910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.671+0000 7f7626cb1f40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5578cd4e4740 0x5578cd4fbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.671+0000 7f7626cb1f40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578cd450950 0x5578cd4fc280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.671+0000 7f7626cb1f40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5578cd3ccf50 con 0x5578cd4e4740 2024-05-09T05:01:15.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.671+0000 7f7626cb1f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5578cd3d0f90 con 0x7f7610003ab0 2024-05-09T05:01:15.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.671+0000 7f7626cb1f40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x5578cd428ef0 con 0x5578cd450950 2024-05-09T05:01:15.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.671+0000 7f761bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5578cd4e4740 0x5578cd4fbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.671+0000 7f761bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5578cd4e4740 0x5578cd4fbd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60046/0 (socket says 172.21.15.60:60046) 2024-05-09T05:01:15.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.671+0000 7f761bfff640 1 -- 172.21.15.60:0/3264837482 learned_addr learned my addr 172.21.15.60:0/3264837482 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:15.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.671+0000 7f761bfff640 1 -- 172.21.15.60:0/3264837482 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578cd450950 msgr2=0x5578cd4fc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:15.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.671+0000 7f761bfff640 1 --2- 172.21.15.60:0/3264837482 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578cd450950 0x5578cd4fc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.671+0000 7f761bfff640 1 -- 172.21.15.60:0/3264837482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7610003ab0 msgr2=0x5578cd454910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:15.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.671+0000 7f761bfff640 1 --2- 172.21.15.60:0/3264837482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7610003ab0 0x5578cd454910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.671+0000 7f761bfff640 1 -- 172.21.15.60:0/3264837482 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578cd4fc7c0 con 0x5578cd4e4740 2024-05-09T05:01:15.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.671+0000 7f761bfff640 1 --2- 172.21.15.60:0/3264837482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5578cd4e4740 0x5578cd4fbd40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f760c000fb0 tx=0x7f760c009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4823dd7d34e2ec9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:15.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.671+0000 7f76237ab640 1 -- 172.21.15.60:0/3264837482 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760c00e8a0 con 0x5578cd4e4740 2024-05-09T05:01:15.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.671+0000 7f76237ab640 1 -- 172.21.15.60:0/3264837482 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f760c013030 con 0x5578cd4e4740 2024-05-09T05:01:15.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.671+0000 7f76237ab640 1 -- 172.21.15.60:0/3264837482 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760c00ec40 con 0x5578cd4e4740 2024-05-09T05:01:15.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.672+0000 7f7626cb1f40 1 -- 172.21.15.60:0/3264837482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5578cd4e4740 msgr2=0x5578cd4fbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:15.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.672+0000 7f7626cb1f40 1 --2- 172.21.15.60:0/3264837482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5578cd4e4740 0x5578cd4fbd40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f760c000fb0 tx=0x7f760c009a50 comp rx=0 tx=0).stop 2024-05-09T05:01:15.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.672+0000 7f7626cb1f40 1 -- 172.21.15.60:0/3264837482 shutdown_connections 2024-05-09T05:01:15.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.672+0000 7f7626cb1f40 1 --2- 172.21.15.60:0/3264837482 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578cd450950 0x5578cd4fc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.672+0000 7f7626cb1f40 1 --2- 172.21.15.60:0/3264837482 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5578cd4e4740 0x5578cd4fbd40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.672+0000 7f7626cb1f40 1 --2- 172.21.15.60:0/3264837482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7610003ab0 0x5578cd454910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.672+0000 7f7626cb1f40 1 -- 172.21.15.60:0/3264837482 >> 172.21.15.60:0/3264837482 conn(0x5578cd452a00 msgr2=0x5578cd4e4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:15.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.672+0000 7f7626cb1f40 1 -- 172.21.15.60:0/3264837482 shutdown_connections 2024-05-09T05:01:15.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.672+0000 7f7626cb1f40 1 -- 172.21.15.60:0/3264837482 wait complete. 2024-05-09T05:01:15.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.673+0000 7f7626cb1f40 1 Processor -- start 2024-05-09T05:01:15.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.673+0000 7f7626cb1f40 1 -- start start 2024-05-09T05:01:15.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.673+0000 7f7626cb1f40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578cd450950 0x5578cd4e17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.673+0000 7f7626cb1f40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5578cd4e4740 0x5578cd4fac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.673+0000 7f7626cb1f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7610003ab0 0x5578cd4fb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.673+0000 7f7626cb1f40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5578cd4e4ee0 con 0x5578cd4e4740 2024-05-09T05:01:15.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.673+0000 7f7626cb1f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5578cd46ebc0 con 0x7f7610003ab0 2024-05-09T05:01:15.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.673+0000 7f7626cb1f40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x5578cd4e5130 con 0x5578cd450950 2024-05-09T05:01:15.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.673+0000 7f761bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5578cd4e4740 0x5578cd4fac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.673+0000 7f761bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5578cd4e4740 0x5578cd4fac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60060/0 (socket says 172.21.15.60:60060) 2024-05-09T05:01:15.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.673+0000 7f761bfff640 1 -- 172.21.15.60:0/979657171 learned_addr learned my addr 172.21.15.60:0/979657171 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:15.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.673+0000 7f7624fae640 1 --2- 172.21.15.60:0/979657171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7610003ab0 0x5578cd4fb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.673+0000 7f76247ad640 1 --2- 172.21.15.60:0/979657171 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578cd450950 0x5578cd4e17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.674+0000 7f761bfff640 1 -- 172.21.15.60:0/979657171 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578cd450950 msgr2=0x5578cd4e17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:15.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.674+0000 7f761bfff640 1 --2- 172.21.15.60:0/979657171 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578cd450950 0x5578cd4e17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.674+0000 7f761bfff640 1 -- 172.21.15.60:0/979657171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7610003ab0 msgr2=0x5578cd4fb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:15.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.674+0000 7f761bfff640 1 --2- 172.21.15.60:0/979657171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7610003ab0 0x5578cd4fb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.674+0000 7f761bfff640 1 -- 172.21.15.60:0/979657171 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578cd4e2100 con 0x5578cd4e4740 2024-05-09T05:01:15.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.674+0000 7f7624fae640 1 --2- 172.21.15.60:0/979657171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7610003ab0 0x5578cd4fb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:15.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.674+0000 7f76247ad640 1 --2- 172.21.15.60:0/979657171 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5578cd450950 0x5578cd4e17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:15.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.674+0000 7f761bfff640 1 --2- 172.21.15.60:0/979657171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5578cd4e4740 0x5578cd4fac20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f760c001220 tx=0x7f760c001a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:15.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.674+0000 7f76227a9640 1 -- 172.21.15.60:0/979657171 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760c00f860 con 0x5578cd4e4740 2024-05-09T05:01:15.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.674+0000 7f76227a9640 1 -- 172.21.15.60:0/979657171 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f760c00fa20 con 0x5578cd4e4740 2024-05-09T05:01:15.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.674+0000 7f7626cb1f40 1 -- 172.21.15.60:0/979657171 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5578cd4e1d20 con 0x5578cd4e4740 2024-05-09T05:01:15.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.674+0000 7f76227a9640 1 -- 172.21.15.60:0/979657171 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760c00fe10 con 0x5578cd4e4740 2024-05-09T05:01:15.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.674+0000 7f7626cb1f40 1 -- 172.21.15.60:0/979657171 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578cd44f730 con 0x5578cd4e4740 2024-05-09T05:01:15.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.676+0000 7f76227a9640 1 -- 172.21.15.60:0/979657171 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f760c019430 con 0x5578cd4e4740 2024-05-09T05:01:15.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.676+0000 7f76227a9640 1 --2- 172.21.15.60:0/979657171 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7604074ce0 0x7f76040771a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.676+0000 7f76227a9640 1 -- 172.21.15.60:0/979657171 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7f760c098550 con 0x5578cd4e4740 2024-05-09T05:01:15.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.676+0000 7f76247ad640 1 --2- 172.21.15.60:0/979657171 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7604074ce0 0x7f76040771a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.676+0000 7f7626cb1f40 1 --2- 172.21.15.60:0/979657171 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x5578cd46a460 0x5578cd46a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:15.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.676+0000 7f7626cb1f40 1 -- 172.21.15.60:0/979657171 --> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] -- osd_op(unknown.0.0:1 6.13 6:ca8fdbba:::100000001fb.00000000:head [getxattr layout in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) -- 0x5578cd46ad80 con 0x5578cd46a460 2024-05-09T05:01:15.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.677+0000 7f7624fae640 1 --2- 172.21.15.60:0/979657171 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x5578cd46a460 0x5578cd46a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:15.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.677+0000 7f76247ad640 1 --2- 172.21.15.60:0/979657171 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7604074ce0 0x7f76040771a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x5578cd58e8a0 tx=0x7f7614003090 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:15.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.677+0000 7f7624fae640 1 --2- 172.21.15.60:0/979657171 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x5578cd46a460 0x5578cd46a840 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:15.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.677+0000 7f7624fae640 1 -- 172.21.15.60:0/979657171 <== osd.3 v2:172.21.15.60:6824/1069806621 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (30) out=30b] v0'0 uv1 ondisk = 0) ==== 164+0+30 (crc 0 0 0) 0x7f761c04d950 con 0x5578cd46a460 2024-05-09T05:01:15.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.677+0000 7f7626cb1f40 1 -- 172.21.15.60:0/979657171 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x5578cd46a460 msgr2=0x5578cd46a840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:15.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.677+0000 7f7626cb1f40 1 --2- 172.21.15.60:0/979657171 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x5578cd46a460 0x5578cd46a840 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:15.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.677+0000 7f7626cb1f40 1 -- 172.21.15.60:0/979657171 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7604074ce0 msgr2=0x7f76040771a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:15.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.677+0000 7f7626cb1f40 1 --2- 172.21.15.60:0/979657171 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7604074ce0 0x7f76040771a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x5578cd58e8a0 tx=0x7f7614003090 comp rx=0 tx=0).stop 2024-05-09T05:01:15.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.677+0000 7f7626cb1f40 1 -- 172.21.15.60:0/979657171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5578cd4e4740 msgr2=0x5578cd4fac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:15.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.677+0000 7f7626cb1f40 1 --2- 172.21.15.60:0/979657171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5578cd4e4740 0x5578cd4fac20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f760c001220 tx=0x7f760c001a60 comp rx=0 tx=0).stop 2024-05-09T05:01:15.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.677+0000 7f7624fae640 1 -- 172.21.15.60:0/979657171 reap_dead start 2024-05-09T05:01:15.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.677+0000 7f7626cb1f40 1 -- 172.21.15.60:0/979657171 shutdown_connections 2024-05-09T05:01:15.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.677+0000 7f7626cb1f40 1 -- 172.21.15.60:0/979657171 >> 172.21.15.60:0/979657171 conn(0x5578cd452a00 msgr2=0x5578cd454910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:15.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.677+0000 7f7626cb1f40 1 -- 172.21.15.60:0/979657171 shutdown_connections 2024-05-09T05:01:15.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.677+0000 7f7626cb1f40 1 -- 172.21.15.60:0/979657171 wait complete. 2024-05-09T05:01:15.683 INFO:teuthology.orchestra.run.smithi060.stdout:@@ 2024-05-09T05:01:15.684 DEBUG:teuthology.orchestra.run.smithi060:> ceph-dencoder type file_layout_t import - decode dump_json 2024-05-09T05:01:15.732 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:01:15.732 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_unit": 4194304, 2024-05-09T05:01:15.732 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_count": 1, 2024-05-09T05:01:15.732 INFO:teuthology.orchestra.run.smithi060.stdout: "object_size": 4194304, 2024-05-09T05:01:15.732 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_id": 6, 2024-05-09T05:01:15.732 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_ns": "" 2024-05-09T05:01:15.732 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:01:15.732 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:15.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:15.734 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent_b) 2024-05-09T05:01:15.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:15.792 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mv parent_a/alpha parent_b/alpha) 2024-05-09T05:01:15.853 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:15.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.996+0000 7f116396a640 1 Processor -- start 2024-05-09T05:01:15.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.997+0000 7f116396a640 1 -- start start 2024-05-09T05:01:16.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.997+0000 7f116396a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f115c0d0070 0x7f115c16a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.997+0000 7f116396a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f115c0d05e0 con 0x7f115c098ed0 2024-05-09T05:01:16.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.997+0000 7f116396a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f115c0ae9b0 con 0x7f115c098af0 2024-05-09T05:01:16.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.997+0000 7f116396a640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f115c09bab0 con 0x7f115c0d0070 2024-05-09T05:01:16.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1160f05640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f115c098ed0 0x7f115c0b2dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33576/0 (socket says 172.21.15.60:33576) 2024-05-09T05:01:16.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1160f05640 1 -- 172.21.15.60:0/3350275852 learned_addr learned my addr 172.21.15.60:0/3350275852 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:16.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1153fff640 1 -- 172.21.15.60:0/3350275852 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1507849015 0 0) 0x7f115c0d05e0 con 0x7f115c098ed0 2024-05-09T05:01:16.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1153fff640 1 -- 172.21.15.60:0/3350275852 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1148003520 con 0x7f115c098ed0 2024-05-09T05:01:16.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1161f07640 1 --2- 172.21.15.60:0/3350275852 >> v2:172.21.15.114:3300/0 conn(0x7f115c0d0070 0x7f115c16a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1153fff640 1 -- 172.21.15.60:0/3350275852 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2802872915 0 0) 0x7f1148003520 con 0x7f115c098ed0 2024-05-09T05:01:16.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1153fff640 1 -- 172.21.15.60:0/3350275852 >> v2:172.21.15.114:3300/0 conn(0x7f115c0d0070 msgr2=0x7f115c16a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:16.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1153fff640 1 --2- 172.21.15.60:0/3350275852 >> v2:172.21.15.114:3300/0 conn(0x7f115c0d0070 0x7f115c16a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1153fff640 1 -- 172.21.15.60:0/3350275852 >> v1:172.21.15.114:6789/0 conn(0x7f115c098af0 legacy=0x7f115c09b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:16.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1153fff640 1 -- 172.21.15.60:0/3350275852 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f115c0aec90 con 0x7f115c098ed0 2024-05-09T05:01:16.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1153fff640 1 -- 172.21.15.60:0/3350275852 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f114c0016f0 con 0x7f115c098ed0 2024-05-09T05:01:16.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1153fff640 1 -- 172.21.15.60:0/3350275852 >> v1:172.21.15.60:6789/0 conn(0x7f115c098ed0 legacy=0x7f115c0b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:16.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1153fff640 1 --2- 172.21.15.60:0/3350275852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1148003a50 0x7f1148003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1153fff640 1 --2- 172.21.15.60:0/3350275852 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1148004800 0x7f1148004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1153fff640 1 --2- 172.21.15.60:0/3350275852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11480051c0 0x7f114800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1153fff640 1 -- 172.21.15.60:0/3350275852 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f115c09bab0 con 0x7f1148003a50 2024-05-09T05:01:16.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1153fff640 1 -- 172.21.15.60:0/3350275852 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f114800c160 con 0x7f11480051c0 2024-05-09T05:01:16.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1153fff640 1 -- 172.21.15.60:0/3350275852 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f114800c360 con 0x7f1148004800 2024-05-09T05:01:16.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1161706640 1 --2- 172.21.15.60:0/3350275852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1148003a50 0x7f1148003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1161f07640 1 --2- 172.21.15.60:0/3350275852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11480051c0 0x7f114800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.998+0000 7f1160f05640 1 --2- 172.21.15.60:0/3350275852 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1148004800 0x7f1148004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.999+0000 7f1161706640 1 -- 172.21.15.60:0/3350275852 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1148004800 msgr2=0x7f1148004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:16.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.999+0000 7f1161706640 1 --2- 172.21.15.60:0/3350275852 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1148004800 0x7f1148004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.999+0000 7f1161706640 1 -- 172.21.15.60:0/3350275852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11480051c0 msgr2=0x7f114800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:16.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.999+0000 7f1161706640 1 --2- 172.21.15.60:0/3350275852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11480051c0 0x7f114800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.999+0000 7f1161706640 1 -- 172.21.15.60:0/3350275852 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1148003520 con 0x7f1148003a50 2024-05-09T05:01:16.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.999+0000 7f1161f07640 1 --2- 172.21.15.60:0/3350275852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11480051c0 0x7f114800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:16.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.999+0000 7f1161f07640 1 -- 172.21.15.60:0/3350275852 reap_dead start 2024-05-09T05:01:16.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.999+0000 7f1161706640 1 --2- 172.21.15.60:0/3350275852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1148003a50 0x7f1148003e70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1158001790 tx=0x7f1148005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94695907cf053817 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:16.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.999+0000 7f1153fff640 1 -- 172.21.15.60:0/3350275852 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11580013b0 con 0x7f1148003a50 2024-05-09T05:01:16.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.999+0000 7f1153fff640 1 -- 172.21.15.60:0/3350275852 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11580052d0 con 0x7f1148003a50 2024-05-09T05:01:16.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:15.999+0000 7f1153fff640 1 -- 172.21.15.60:0/3350275852 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115802aa90 con 0x7f1148003a50 2024-05-09T05:01:16.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.000+0000 7f116396a640 1 -- 172.21.15.60:0/3350275852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1148003a50 msgr2=0x7f1148003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:16.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.000+0000 7f116396a640 1 --2- 172.21.15.60:0/3350275852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1148003a50 0x7f1148003e70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1158001790 tx=0x7f1148005830 comp rx=0 tx=0).stop 2024-05-09T05:01:16.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.000+0000 7f116396a640 1 -- 172.21.15.60:0/3350275852 shutdown_connections 2024-05-09T05:01:16.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.000+0000 7f116396a640 1 --2- 172.21.15.60:0/3350275852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1148003a50 0x7f1148003e70 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.000+0000 7f116396a640 1 -- 172.21.15.60:0/3350275852 >> 172.21.15.60:0/3350275852 conn(0x7f115c0b42e0 msgr2=0x7f115c0b46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:16.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.000+0000 7f116396a640 1 -- 172.21.15.60:0/3350275852 shutdown_connections 2024-05-09T05:01:16.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.000+0000 7f116396a640 1 -- 172.21.15.60:0/3350275852 wait complete. 2024-05-09T05:01:16.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.001+0000 7f116396a640 1 Processor -- start 2024-05-09T05:01:16.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.001+0000 7f116396a640 1 -- start start 2024-05-09T05:01:16.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.002+0000 7f116396a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1148003a50 0x7f115c163b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.002+0000 7f116396a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f115c15db90 0x7f115c15d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.002+0000 7f116396a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f115c161ea0 0x7f115c162350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.002+0000 7f116396a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f115c0b17e0 con 0x7f115c15db90 2024-05-09T05:01:16.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.002+0000 7f116396a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f115c0b28a0 con 0x7f115c161ea0 2024-05-09T05:01:16.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.002+0000 7f116396a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f115c0b1960 con 0x7f1148003a50 2024-05-09T05:01:16.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.002+0000 7f1160f05640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f115c15db90 0x7f115c15d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.002+0000 7f1160f05640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f115c15db90 0x7f115c15d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60076/0 (socket says 172.21.15.60:60076) 2024-05-09T05:01:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.002+0000 7f1160f05640 1 -- 172.21.15.60:0/231225074 learned_addr learned my addr 172.21.15.60:0/231225074 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.002+0000 7f1160f05640 1 -- 172.21.15.60:0/231225074 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1148003a50 msgr2=0x7f115c163b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.002+0000 7f1160f05640 1 --2- 172.21.15.60:0/231225074 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1148003a50 0x7f115c163b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.002+0000 7f1160f05640 1 -- 172.21.15.60:0/231225074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f115c161ea0 msgr2=0x7f115c162350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.002+0000 7f1160f05640 1 --2- 172.21.15.60:0/231225074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f115c161ea0 0x7f115c162350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.002+0000 7f1160f05640 1 -- 172.21.15.60:0/231225074 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f115c15e010 con 0x7f115c15db90 2024-05-09T05:01:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.003+0000 7f1160f05640 1 --2- 172.21.15.60:0/231225074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f115c15db90 0x7f115c15d3e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f114c00e780 tx=0x7f114c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.003+0000 7f1151ffb640 1 -- 172.21.15.60:0/231225074 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c00ee90 con 0x7f115c15db90 2024-05-09T05:01:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.003+0000 7f1151ffb640 1 -- 172.21.15.60:0/231225074 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f114c018070 con 0x7f115c15db90 2024-05-09T05:01:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.003+0000 7f116396a640 1 -- 172.21.15.60:0/231225074 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f115c15e2c0 con 0x7f115c15db90 2024-05-09T05:01:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.003+0000 7f1151ffb640 1 -- 172.21.15.60:0/231225074 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c00ca90 con 0x7f115c15db90 2024-05-09T05:01:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.003+0000 7f116396a640 1 -- 172.21.15.60:0/231225074 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f115c0aaeb0 con 0x7f115c15db90 2024-05-09T05:01:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.004+0000 7f116396a640 1 -- 172.21.15.60:0/231225074 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f115c0b4bf0 con 0x7f115c15db90 2024-05-09T05:01:16.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.004+0000 7f1151ffb640 1 -- 172.21.15.60:0/231225074 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f114c00cc30 con 0x7f115c15db90 2024-05-09T05:01:16.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.005+0000 7f1151ffb640 1 --2- 172.21.15.60:0/231225074 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f11400764a0 0x7f1140078960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.005+0000 7f1151ffb640 1 -- 172.21.15.60:0/231225074 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7f114c0971e0 con 0x7f115c15db90 2024-05-09T05:01:16.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.005+0000 7f1161706640 1 --2- 172.21.15.60:0/231225074 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f11400764a0 0x7f1140078960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.006+0000 7f1161706640 1 --2- 172.21.15.60:0/231225074 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f11400764a0 0x7f1140078960 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1158032000 tx=0x7f115802c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:16.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.009+0000 7f1151ffb640 1 -- 172.21.15.60:0/231225074 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f115c0b4bf0 con 0x7f115c15db90 2024-05-09T05:01:16.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.238+0000 7f116396a640 1 -- 172.21.15.60:0/231225074 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f115c0aaca0 con 0x7f115c15db90 2024-05-09T05:01:16.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.238+0000 7f1151ffb640 1 -- 172.21.15.60:0/231225074 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5181 (secure 0 0 0) 0x7f114c0609b0 con 0x7f115c15db90 2024-05-09T05:01:16.239 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:16.239 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4547,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2372252858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2372252858},{"type":"v1","addr":"172.21.15.114:6836","nonce":2372252858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1761282610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1761282610},{"type":"v1","addr":"172.21.15.114:6837","nonce":1761282610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4765,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/300408012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":300408012},{"type":"v1","addr":"172.21.15.60:6835","nonce":300408012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:04.698975+0000","modified":"2024-05-09T05:01:07.722167+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4893},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4893":{"gid":4893,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/399160914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":399160914},{"type":"v1","addr":"172.21.15.60:6834","nonce":399160914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4893,"qdb_cluster":[4893]},"id":2}]} 2024-05-09T05:01:16.240 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 17 2024-05-09T05:01:16.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.241+0000 7f116396a640 1 -- 172.21.15.60:0/231225074 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f11400764a0 msgr2=0x7f1140078960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:16.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.241+0000 7f116396a640 1 --2- 172.21.15.60:0/231225074 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f11400764a0 0x7f1140078960 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1158032000 tx=0x7f115802c000 comp rx=0 tx=0).stop 2024-05-09T05:01:16.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.241+0000 7f116396a640 1 -- 172.21.15.60:0/231225074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f115c15db90 msgr2=0x7f115c15d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:16.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.241+0000 7f116396a640 1 --2- 172.21.15.60:0/231225074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f115c15db90 0x7f115c15d3e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f114c00e780 tx=0x7f114c00ec40 comp rx=0 tx=0).stop 2024-05-09T05:01:16.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.241+0000 7f116396a640 1 -- 172.21.15.60:0/231225074 shutdown_connections 2024-05-09T05:01:16.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.241+0000 7f116396a640 1 --2- 172.21.15.60:0/231225074 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f11400764a0 0x7f1140078960 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.241+0000 7f116396a640 1 --2- 172.21.15.60:0/231225074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f115c161ea0 0x7f115c162350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.241+0000 7f116396a640 1 --2- 172.21.15.60:0/231225074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f115c15db90 0x7f115c15d3e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.241+0000 7f116396a640 1 --2- 172.21.15.60:0/231225074 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1148003a50 0x7f115c163b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.241+0000 7f116396a640 1 -- 172.21.15.60:0/231225074 >> 172.21.15.60:0/231225074 conn(0x7f115c0b42e0 msgr2=0x7f115c0d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:16.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.241+0000 7f116396a640 1 -- 172.21.15.60:0/231225074 shutdown_connections 2024-05-09T05:01:16.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.241+0000 7f116396a640 1 -- 172.21.15.60:0/231225074 wait complete. 2024-05-09T05:01:16.252 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-05-09T05:01:16.381 INFO:teuthology.orchestra.run.smithi060.stdout:{"message":"","return_code":0} 2024-05-09T05:01:16.381 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-05-09T05:01:16.381 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001fb.00000000 parent 2024-05-09T05:01:16.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.426+0000 7f71e323bf40 1 Processor -- start 2024-05-09T05:01:16.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.426+0000 7f71e323bf40 1 -- start start 2024-05-09T05:01:16.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.426+0000 7f71e323bf40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71cc003ab0 0x5653e3ecd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.426+0000 7f71e323bf40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5653e3ebf850 0x5653e3ebeb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.426+0000 7f71e323bf40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5653e3e46810 0x5653e3ed7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.426+0000 7f71e323bf40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5653e3da7f50 con 0x5653e3ebf850 2024-05-09T05:01:16.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.426+0000 7f71e323bf40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5653e3dabf90 con 0x7f71cc003ab0 2024-05-09T05:01:16.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.426+0000 7f71e323bf40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x5653e3ed5b40 con 0x5653e3e46810 2024-05-09T05:01:16.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.427+0000 7f71e0536640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5653e3ebf850 0x5653e3ebeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.427+0000 7f71e0536640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5653e3ebf850 0x5653e3ebeb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60090/0 (socket says 172.21.15.60:60090) 2024-05-09T05:01:16.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.427+0000 7f71e0536640 1 -- 172.21.15.60:0/72970993 learned_addr learned my addr 172.21.15.60:0/72970993 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:16.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.427+0000 7f71e0d37640 1 --2- 172.21.15.60:0/72970993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71cc003ab0 0x5653e3ecd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.427+0000 7f71e1538640 1 --2- 172.21.15.60:0/72970993 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5653e3e46810 0x5653e3ed7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.427+0000 7f71e0536640 1 -- 172.21.15.60:0/72970993 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5653e3e46810 msgr2=0x5653e3ed7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:16.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.427+0000 7f71e0536640 1 --2- 172.21.15.60:0/72970993 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5653e3e46810 0x5653e3ed7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.427+0000 7f71e0536640 1 -- 172.21.15.60:0/72970993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71cc003ab0 msgr2=0x5653e3ecd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:16.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.427+0000 7f71e0536640 1 --2- 172.21.15.60:0/72970993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71cc003ab0 0x5653e3ecd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.427+0000 7f71e0536640 1 -- 172.21.15.60:0/72970993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653e3ebf3a0 con 0x5653e3ebf850 2024-05-09T05:01:16.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.427+0000 7f71e0d37640 1 --2- 172.21.15.60:0/72970993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71cc003ab0 0x5653e3ecd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:16.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.427+0000 7f71e0536640 1 --2- 172.21.15.60:0/72970993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5653e3ebf850 0x5653e3ebeb40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f71c800d6f0 tx=0x7f71c800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b146505a39740b0b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:16.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.427+0000 7f71df534640 1 -- 172.21.15.60:0/72970993 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c8013070 con 0x5653e3ebf850 2024-05-09T05:01:16.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.427+0000 7f71df534640 1 -- 172.21.15.60:0/72970993 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c800b870 con 0x5653e3ebf850 2024-05-09T05:01:16.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.427+0000 7f71df534640 1 -- 172.21.15.60:0/72970993 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c800bc20 con 0x5653e3ebf850 2024-05-09T05:01:16.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.428+0000 7f71e323bf40 1 -- 172.21.15.60:0/72970993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5653e3ebf850 msgr2=0x5653e3ebeb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:16.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.428+0000 7f71e323bf40 1 --2- 172.21.15.60:0/72970993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5653e3ebf850 0x5653e3ebeb40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f71c800d6f0 tx=0x7f71c800dbb0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.428+0000 7f71e323bf40 1 -- 172.21.15.60:0/72970993 shutdown_connections 2024-05-09T05:01:16.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.428+0000 7f71e323bf40 1 --2- 172.21.15.60:0/72970993 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5653e3e46810 0x5653e3ed7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.428+0000 7f71e323bf40 1 --2- 172.21.15.60:0/72970993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5653e3ebf850 0x5653e3ebeb40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.428+0000 7f71e323bf40 1 --2- 172.21.15.60:0/72970993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f71cc003ab0 0x5653e3ecd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.428+0000 7f71e323bf40 1 -- 172.21.15.60:0/72970993 >> 172.21.15.60:0/72970993 conn(0x5653e3e00110 msgr2=0x5653e3e48420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:16.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.428+0000 7f71e323bf40 1 -- 172.21.15.60:0/72970993 shutdown_connections 2024-05-09T05:01:16.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.428+0000 7f71e323bf40 1 -- 172.21.15.60:0/72970993 wait complete. 2024-05-09T05:01:16.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.428+0000 7f71e323bf40 1 Processor -- start 2024-05-09T05:01:16.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.428+0000 7f71e323bf40 1 -- start start 2024-05-09T05:01:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.429+0000 7f71e323bf40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653e3e46810 0x5653e3ed77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.429+0000 7f71e323bf40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5653e3ebf850 0x5653e3ebb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.429+0000 7f71e323bf40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71cc003ab0 0x5653e3ed7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.429+0000 7f71e323bf40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5653e3ebf680 con 0x5653e3ebf850 2024-05-09T05:01:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.429+0000 7f71e323bf40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5653e3ed4c10 con 0x5653e3e46810 2024-05-09T05:01:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.429+0000 7f71e323bf40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x5653e3e00590 con 0x7f71cc003ab0 2024-05-09T05:01:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.429+0000 7f71e0536640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5653e3ebf850 0x5653e3ebb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.429+0000 7f71e0536640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5653e3ebf850 0x5653e3ebb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60106/0 (socket says 172.21.15.60:60106) 2024-05-09T05:01:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.429+0000 7f71e0536640 1 -- 172.21.15.60:0/1929793024 learned_addr learned my addr 172.21.15.60:0/1929793024 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.429+0000 7f71e0536640 1 -- 172.21.15.60:0/1929793024 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71cc003ab0 msgr2=0x5653e3ed7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.429+0000 7f71e0d37640 1 --2- 172.21.15.60:0/1929793024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653e3e46810 0x5653e3ed77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.429+0000 7f71e1538640 1 --2- 172.21.15.60:0/1929793024 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71cc003ab0 0x5653e3ed7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.429+0000 7f71e0536640 1 --2- 172.21.15.60:0/1929793024 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71cc003ab0 0x5653e3ed7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.429+0000 7f71e0536640 1 -- 172.21.15.60:0/1929793024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653e3e46810 msgr2=0x5653e3ed77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.429+0000 7f71e0536640 1 --2- 172.21.15.60:0/1929793024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653e3e46810 0x5653e3ed77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.429+0000 7f71e0536640 1 -- 172.21.15.60:0/1929793024 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653e3ebd1d0 con 0x5653e3ebf850 2024-05-09T05:01:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.429+0000 7f71e0d37640 1 --2- 172.21.15.60:0/1929793024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5653e3e46810 0x5653e3ed77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.429+0000 7f71e1538640 1 --2- 172.21.15.60:0/1929793024 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f71cc003ab0 0x5653e3ed7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.430+0000 7f71e0536640 1 --2- 172.21.15.60:0/1929793024 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5653e3ebf850 0x5653e3ebb750 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f71c800d6c0 tx=0x5653e3e2a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.430+0000 7f71de532640 1 -- 172.21.15.60:0/1929793024 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c8028070 con 0x5653e3ebf850 2024-05-09T05:01:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.430+0000 7f71de532640 1 -- 172.21.15.60:0/1929793024 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c8018cb0 con 0x5653e3ebf850 2024-05-09T05:01:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.430+0000 7f71e323bf40 1 -- 172.21.15.60:0/1929793024 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5653e3ebcbb0 con 0x5653e3ebf850 2024-05-09T05:01:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.430+0000 7f71de532640 1 -- 172.21.15.60:0/1929793024 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c8013050 con 0x5653e3ebf850 2024-05-09T05:01:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.430+0000 7f71e323bf40 1 -- 172.21.15.60:0/1929793024 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653e3ebce90 con 0x5653e3ebf850 2024-05-09T05:01:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.431+0000 7f71de532640 1 -- 172.21.15.60:0/1929793024 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f71c802b070 con 0x5653e3ebf850 2024-05-09T05:01:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.432+0000 7f71de532640 1 --2- 172.21.15.60:0/1929793024 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f71c0074cb0 0x7f71c0077170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.432+0000 7f71de532640 1 -- 172.21.15.60:0/1929793024 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7f71c80981e0 con 0x5653e3ebf850 2024-05-09T05:01:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.432+0000 7f71e323bf40 1 --2- 172.21.15.60:0/1929793024 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x5653e3f6c720 0x5653e3f6cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.432+0000 7f71e323bf40 1 -- 172.21.15.60:0/1929793024 --> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] -- osd_op(unknown.0.0:1 6.13 6:ca8fdbba:::100000001fb.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36) -- 0x5653e3f6d0f0 con 0x5653e3f6c720 2024-05-09T05:01:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.432+0000 7f71e0d37640 1 --2- 172.21.15.60:0/1929793024 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f71c0074cb0 0x7f71c0077170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.432+0000 7f71e1538640 1 --2- 172.21.15.60:0/1929793024 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x5653e3f6c720 0x5653e3f6cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.433+0000 7f71e1538640 1 --2- 172.21.15.60:0/1929793024 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x5653e3f6c720 0x5653e3f6cbb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.433+0000 7f71e0d37640 1 --2- 172.21.15.60:0/1929793024 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f71c0074cb0 0x7f71c0077170 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x5653e3e02c10 tx=0x7f71d800c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.433+0000 7f71e1538640 1 -- 172.21.15.60:0/1929793024 <== osd.3 v2:172.21.15.60:6824/1069806621 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (95) out=95b] v0'0 uv2 ondisk = 0) ==== 164+0+95 (crc 0 0 0) 0x7f71d005a040 con 0x5653e3f6c720 2024-05-09T05:01:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.433+0000 7f71e323bf40 1 -- 172.21.15.60:0/1929793024 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x5653e3f6c720 msgr2=0x5653e3f6cbb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.433+0000 7f71e323bf40 1 --2- 172.21.15.60:0/1929793024 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x5653e3f6c720 0x5653e3f6cbb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.433+0000 7f71e323bf40 1 -- 172.21.15.60:0/1929793024 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f71c0074cb0 msgr2=0x7f71c0077170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.433+0000 7f71e323bf40 1 --2- 172.21.15.60:0/1929793024 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f71c0074cb0 0x7f71c0077170 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x5653e3e02c10 tx=0x7f71d800c040 comp rx=0 tx=0).stop 2024-05-09T05:01:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.433+0000 7f71e323bf40 1 -- 172.21.15.60:0/1929793024 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5653e3ebf850 msgr2=0x5653e3ebb750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.433+0000 7f71e323bf40 1 --2- 172.21.15.60:0/1929793024 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5653e3ebf850 0x5653e3ebb750 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f71c800d6c0 tx=0x5653e3e2a640 comp rx=0 tx=0).stop 2024-05-09T05:01:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.433+0000 7f71e1538640 1 -- 172.21.15.60:0/1929793024 reap_dead start 2024-05-09T05:01:16.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.433+0000 7f71e323bf40 1 -- 172.21.15.60:0/1929793024 shutdown_connections 2024-05-09T05:01:16.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.433+0000 7f71e323bf40 1 -- 172.21.15.60:0/1929793024 >> 172.21.15.60:0/1929793024 conn(0x5653e3e00110 msgr2=0x5653e3ebeb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:16.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.433+0000 7f71e323bf40 1 -- 172.21.15.60:0/1929793024 shutdown_connections 2024-05-09T05:01:16.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.433+0000 7f71e323bf40 1 -- 172.21.15.60:0/1929793024 wait complete. 2024-05-09T05:01:16.438 INFO:teuthology.orchestra.run.smithi060.stdout:Y��alphaparent_b 2024-05-09T05:01:16.439 DEBUG:teuthology.orchestra.run.smithi060:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-05-09T05:01:16.487 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:01:16.487 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628283, 2024-05-09T05:01:16.487 INFO:teuthology.orchestra.run.smithi060.stdout: "ancestors": [ 2024-05-09T05:01:16.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:16.487 INFO:teuthology.orchestra.run.smithi060.stdout: "dirino": 1099511628284, 2024-05-09T05:01:16.487 INFO:teuthology.orchestra.run.smithi060.stdout: "dname": "alpha", 2024-05-09T05:01:16.487 INFO:teuthology.orchestra.run.smithi060.stdout: "version": 3 2024-05-09T05:01:16.488 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:16.488 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:16.488 INFO:teuthology.orchestra.run.smithi060.stdout: "dirino": 1, 2024-05-09T05:01:16.488 INFO:teuthology.orchestra.run.smithi060.stdout: "dname": "parent_b", 2024-05-09T05:01:16.488 INFO:teuthology.orchestra.run.smithi060.stdout: "version": 25 2024-05-09T05:01:16.488 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:16.488 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-05-09T05:01:16.488 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 6, 2024-05-09T05:01:16.488 INFO:teuthology.orchestra.run.smithi060.stdout: "old_pools": [] 2024-05-09T05:01:16.488 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:01:16.488 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:16.489 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create data_new --pg_num_min 64 2024-05-09T05:01:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.637+0000 7fa0bee75640 1 Processor -- start 2024-05-09T05:01:16.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.637+0000 7fa0bee75640 1 -- start start 2024-05-09T05:01:16.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.638+0000 7fa0bee75640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0b80d4cc0 0x7fa0b80b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.638+0000 7fa0bee75640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa0b8148710 0x7fa0b80d1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.638+0000 7fa0bee75640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0b80b5a90 con 0x7fa0b81481b0 2024-05-09T05:01:16.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.638+0000 7fa0bee75640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fa0b80b5d70 con 0x7fa0b8148710 2024-05-09T05:01:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.638+0000 7fa0bee75640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa0b80b6050 con 0x7fa0b80d4cc0 2024-05-09T05:01:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.638+0000 7fa0bd412640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa0b8148710 0x7fa0b80d1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.638+0000 7fa0bd412640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa0b8148710 0x7fa0b80d1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60110/0 (socket says 172.21.15.60:60110) 2024-05-09T05:01:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.638+0000 7fa0bd412640 1 -- 172.21.15.60:0/280265801 learned_addr learned my addr 172.21.15.60:0/280265801 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.639+0000 7fa0b7fff640 1 --2- 172.21.15.60:0/280265801 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0b80d4cc0 0x7fa0b80b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.639+0000 7fa0bd412640 1 -- 172.21.15.60:0/280265801 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0b80d4cc0 msgr2=0x7fa0b80b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.639+0000 7fa0bd412640 1 --2- 172.21.15.60:0/280265801 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0b80d4cc0 0x7fa0b80b54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.639+0000 7fa0bd412640 1 -- 172.21.15.60:0/280265801 >> v1:172.21.15.114:6789/0 conn(0x7fa0b81481b0 legacy=0x7fa0b80d45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.639+0000 7fa0bd412640 1 -- 172.21.15.60:0/280265801 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b80b6330 con 0x7fa0b8148710 2024-05-09T05:01:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.639+0000 7fa0bd412640 1 --2- 172.21.15.60:0/280265801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0b8148710 0x7fa0b80d1d10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0009e50 tx=0x7fa0b002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b5c0758605dbcfb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.640+0000 7fa0b77fe640 1 -- 172.21.15.60:0/280265801 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0036070 con 0x7fa0b8148710 2024-05-09T05:01:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.640+0000 7fa0b77fe640 1 -- 172.21.15.60:0/280265801 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0b0031600 con 0x7fa0b8148710 2024-05-09T05:01:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.640+0000 7fa0b77fe640 1 -- 172.21.15.60:0/280265801 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b00319d0 con 0x7fa0b8148710 2024-05-09T05:01:16.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.640+0000 7fa0bee75640 1 -- 172.21.15.60:0/280265801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0b8148710 msgr2=0x7fa0b80d1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.640+0000 7fa0bee75640 1 --2- 172.21.15.60:0/280265801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0b8148710 0x7fa0b80d1d10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0009e50 tx=0x7fa0b002a5f0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.640+0000 7fa0bee75640 1 -- 172.21.15.60:0/280265801 shutdown_connections 2024-05-09T05:01:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.640+0000 7fa0bee75640 1 --2- 172.21.15.60:0/280265801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0b8148710 0x7fa0b80d1d10 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.640+0000 7fa0bee75640 1 --2- 172.21.15.60:0/280265801 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0b80d4cc0 0x7fa0b80b54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.640+0000 7fa0bee75640 1 -- 172.21.15.60:0/280265801 >> 172.21.15.60:0/280265801 conn(0x7fa0b806f640 msgr2=0x7fa0b8073840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.640+0000 7fa0bee75640 1 -- 172.21.15.60:0/280265801 shutdown_connections 2024-05-09T05:01:16.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.640+0000 7fa0bee75640 1 -- 172.21.15.60:0/280265801 wait complete. 2024-05-09T05:01:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.641+0000 7fa0bee75640 1 Processor -- start 2024-05-09T05:01:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.642+0000 7fa0bee75640 1 -- start start 2024-05-09T05:01:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.642+0000 7fa0bee75640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0b80d4cc0 0x7fa0b8160de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.642+0000 7fa0bee75640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0b81481b0 0x7fa0b813eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.642+0000 7fa0bee75640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0b8148710 0x7fa0b813ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.642+0000 7fa0bee75640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b80d23d0 con 0x7fa0b81481b0 2024-05-09T05:01:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.642+0000 7fa0bee75640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b80d2250 con 0x7fa0b80d4cc0 2024-05-09T05:01:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.642+0000 7fa0bee75640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa0b80d2550 con 0x7fa0b8148710 2024-05-09T05:01:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.643+0000 7fa0b7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0b81481b0 0x7fa0b813eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.643+0000 7fa0b7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0b81481b0 0x7fa0b813eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60114/0 (socket says 172.21.15.60:60114) 2024-05-09T05:01:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.643+0000 7fa0b7fff640 1 -- 172.21.15.60:0/2810628412 learned_addr learned my addr 172.21.15.60:0/2810628412 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.643+0000 7fa0bcc11640 1 --2- 172.21.15.60:0/2810628412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0b80d4cc0 0x7fa0b8160de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.643+0000 7fa0bd412640 1 --2- 172.21.15.60:0/2810628412 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0b8148710 0x7fa0b813ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.643+0000 7fa0b7fff640 1 -- 172.21.15.60:0/2810628412 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0b8148710 msgr2=0x7fa0b813ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.643+0000 7fa0b7fff640 1 --2- 172.21.15.60:0/2810628412 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0b8148710 0x7fa0b813ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.643+0000 7fa0b7fff640 1 -- 172.21.15.60:0/2810628412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0b80d4cc0 msgr2=0x7fa0b8160de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.643+0000 7fa0b7fff640 1 --2- 172.21.15.60:0/2810628412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0b80d4cc0 0x7fa0b8160de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.643+0000 7fa0b7fff640 1 -- 172.21.15.60:0/2810628412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b813f080 con 0x7fa0b81481b0 2024-05-09T05:01:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.643+0000 7fa0b7fff640 1 --2- 172.21.15.60:0/2810628412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0b81481b0 0x7fa0b813eb40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400c7f0 tx=0x7fa0a400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.643+0000 7fa0b57fa640 1 -- 172.21.15.60:0/2810628412 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4015070 con 0x7fa0b81481b0 2024-05-09T05:01:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.643+0000 7fa0bee75640 1 -- 172.21.15.60:0/2810628412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0b81404d0 con 0x7fa0b81481b0 2024-05-09T05:01:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.643+0000 7fa0bee75640 1 -- 172.21.15.60:0/2810628412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0b8140a50 con 0x7fa0b81481b0 2024-05-09T05:01:16.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.643+0000 7fa0b57fa640 1 -- 172.21.15.60:0/2810628412 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0a4004430 con 0x7fa0b81481b0 2024-05-09T05:01:16.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.643+0000 7fa0b57fa640 1 -- 172.21.15.60:0/2810628412 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a401e060 con 0x7fa0b81481b0 2024-05-09T05:01:16.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.644+0000 7fa0bee75640 1 -- 172.21.15.60:0/2810628412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa098005290 con 0x7fa0b81481b0 2024-05-09T05:01:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.645+0000 7fa0b57fa640 1 -- 172.21.15.60:0/2810628412 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa0a4003810 con 0x7fa0b81481b0 2024-05-09T05:01:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.645+0000 7fa0b57fa640 1 --2- 172.21.15.60:0/2810628412 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa09c0762d0 0x7fa09c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.646+0000 7fa0b57fa640 1 -- 172.21.15.60:0/2810628412 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6101+0+0 (secure 0 0 0) 0x7fa0a4097860 con 0x7fa0b81481b0 2024-05-09T05:01:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.646+0000 7fa0bcc11640 1 --2- 172.21.15.60:0/2810628412 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa09c0762d0 0x7fa09c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.646+0000 7fa0bcc11640 1 --2- 172.21.15.60:0/2810628412 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa09c0762d0 0x7fa09c078790 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac004f60 tx=0x7fa0ac007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:16.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.650+0000 7fa0b57fa640 1 -- 172.21.15.60:0/2810628412 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa0b813f080 con 0x7fa0b81481b0 2024-05-09T05:01:16.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:16.842+0000 7fa0bee75640 1 -- 172.21.15.60:0/2810628412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64} v 0) -- 0x7fa098005d90 con 0x7fa0b81481b0 2024-05-09T05:01:17.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.764+0000 7fa0b57fa640 1 -- 172.21.15.60:0/2810628412 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64}]=0 pool 'data_new' created v37) ==== 124+0+0 (secure 0 0 0) 0x7fa0a4060830 con 0x7fa0b81481b0 2024-05-09T05:01:17.765 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'data_new' created 2024-05-09T05:01:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.767+0000 7fa0bee75640 1 -- 172.21.15.60:0/2810628412 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa09c0762d0 msgr2=0x7fa09c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.767+0000 7fa0bee75640 1 --2- 172.21.15.60:0/2810628412 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa09c0762d0 0x7fa09c078790 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac004f60 tx=0x7fa0ac007e30 comp rx=0 tx=0).stop 2024-05-09T05:01:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.767+0000 7fa0bee75640 1 -- 172.21.15.60:0/2810628412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0b81481b0 msgr2=0x7fa0b813eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.767+0000 7fa0bee75640 1 --2- 172.21.15.60:0/2810628412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0b81481b0 0x7fa0b813eb40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400c7f0 tx=0x7fa0a400ccb0 comp rx=0 tx=0).stop 2024-05-09T05:01:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.767+0000 7fa0bee75640 1 -- 172.21.15.60:0/2810628412 shutdown_connections 2024-05-09T05:01:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.767+0000 7fa0bee75640 1 --2- 172.21.15.60:0/2810628412 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa09c0762d0 0x7fa09c078790 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.767+0000 7fa0bee75640 1 --2- 172.21.15.60:0/2810628412 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa0b8148710 0x7fa0b813ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.767+0000 7fa0bee75640 1 --2- 172.21.15.60:0/2810628412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0b81481b0 0x7fa0b813eb40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.767+0000 7fa0bee75640 1 --2- 172.21.15.60:0/2810628412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0b80d4cc0 0x7fa0b8160de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.767+0000 7fa0bee75640 1 -- 172.21.15.60:0/2810628412 >> 172.21.15.60:0/2810628412 conn(0x7fa0b806f640 msgr2=0x7fa0b80d29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:17.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.768+0000 7fa0bee75640 1 -- 172.21.15.60:0/2810628412 shutdown_connections 2024-05-09T05:01:17.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.768+0000 7fa0bee75640 1 -- 172.21.15.60:0/2810628412 wait complete. 2024-05-09T05:01:17.787 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs data_new 2024-05-09T05:01:17.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.898+0000 7f678fb32640 1 Processor -- start 2024-05-09T05:01:17.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.899+0000 7f678fb32640 1 -- start start 2024-05-09T05:01:17.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.899+0000 7f678fb32640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67880d1050 0x7f67880ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:17.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.899+0000 7f678fb32640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f67880d1430 0x7f67880d18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:17.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.899+0000 7f678fb32640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67880af370 con 0x7f6788096eb0 2024-05-09T05:01:17.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.899+0000 7f678fb32640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f6788096a00 con 0x7f67880d1430 2024-05-09T05:01:17.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.899+0000 7f678fb32640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f67880d2dd0 con 0x7f67880d1050 2024-05-09T05:01:17.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.899+0000 7f678e0cf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f67880d1430 0x7f67880d18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:17.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.899+0000 7f678d0cd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6788096eb0 0x7f678809b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33586/0 (socket says 172.21.15.60:33586) 2024-05-09T05:01:17.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.900+0000 7f678d0cd640 1 -- 172.21.15.60:0/3149528364 learned_addr learned my addr 172.21.15.60:0/3149528364 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:17.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.900+0000 7f678d8ce640 1 --2- 172.21.15.60:0/3149528364 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67880d1050 0x7f67880ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:17.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.900+0000 7f678e0cf640 1 -- 172.21.15.60:0/3149528364 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67880d1050 msgr2=0x7f67880ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:17.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.900+0000 7f678e0cf640 1 --2- 172.21.15.60:0/3149528364 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67880d1050 0x7f67880ae9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:17.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.900+0000 7f678e0cf640 1 -- 172.21.15.60:0/3149528364 >> v1:172.21.15.60:6789/0 conn(0x7f6788096eb0 legacy=0x7f678809b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:17.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.900+0000 7f678c8cc640 1 -- 172.21.15.60:0/3149528364 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2072484652 0 0) 0x7f67880af370 con 0x7f6788096eb0 2024-05-09T05:01:17.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.900+0000 7f678e0cf640 1 -- 172.21.15.60:0/3149528364 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67880d30b0 con 0x7f67880d1430 2024-05-09T05:01:17.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.901+0000 7f678e0cf640 1 --2- 172.21.15.60:0/3149528364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67880d1430 0x7f67880d18e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6784009d90 tx=0x7f678402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b01f942611f306d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:17.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.901+0000 7f678c8cc640 1 -- 172.21.15.60:0/3149528364 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6784007740 con 0x7f67880d1430 2024-05-09T05:01:17.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.901+0000 7f678c8cc640 1 -- 172.21.15.60:0/3149528364 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67840078e0 con 0x7f67880d1430 2024-05-09T05:01:17.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.901+0000 7f678c8cc640 1 -- 172.21.15.60:0/3149528364 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6784007cb0 con 0x7f67880d1430 2024-05-09T05:01:17.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.901+0000 7f678fb32640 1 -- 172.21.15.60:0/3149528364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67880d1430 msgr2=0x7f67880d18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:17.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.901+0000 7f678fb32640 1 --2- 172.21.15.60:0/3149528364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67880d1430 0x7f67880d18e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6784009d90 tx=0x7f678402a660 comp rx=0 tx=0).stop 2024-05-09T05:01:17.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.901+0000 7f678fb32640 1 -- 172.21.15.60:0/3149528364 shutdown_connections 2024-05-09T05:01:17.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.901+0000 7f678fb32640 1 --2- 172.21.15.60:0/3149528364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67880d1430 0x7f67880d18e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:17.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.901+0000 7f678fb32640 1 --2- 172.21.15.60:0/3149528364 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67880d1050 0x7f67880ae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:17.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.901+0000 7f678fb32640 1 -- 172.21.15.60:0/3149528364 >> 172.21.15.60:0/3149528364 conn(0x7f67880993f0 msgr2=0x7f6788099800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:17.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.901+0000 7f678fb32640 1 -- 172.21.15.60:0/3149528364 shutdown_connections 2024-05-09T05:01:17.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.901+0000 7f678fb32640 1 -- 172.21.15.60:0/3149528364 wait complete. 2024-05-09T05:01:17.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.903+0000 7f678fb32640 1 Processor -- start 2024-05-09T05:01:17.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.903+0000 7f678fb32640 1 -- start start 2024-05-09T05:01:17.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.903+0000 7f678fb32640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6788096eb0 0x7f678806fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:17.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.903+0000 7f678fb32640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67880d1050 0x7f678806dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:17.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.903+0000 7f678fb32640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67880d1430 0x7f67880913c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:17.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.903+0000 7f678fb32640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f678809c1e0 con 0x7f67880d1050 2024-05-09T05:01:17.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.903+0000 7f678fb32640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f678809c060 con 0x7f6788096eb0 2024-05-09T05:01:17.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.903+0000 7f678fb32640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f67880d2790 con 0x7f67880d1430 2024-05-09T05:01:17.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.903+0000 7f678d0cd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67880d1050 0x7f678806dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:17.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.903+0000 7f678d0cd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67880d1050 0x7f678806dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60140/0 (socket says 172.21.15.60:60140) 2024-05-09T05:01:17.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.904+0000 7f678d0cd640 1 -- 172.21.15.60:0/2572198602 learned_addr learned my addr 172.21.15.60:0/2572198602 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:17.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.904+0000 7f678e0cf640 1 --2- 172.21.15.60:0/2572198602 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67880d1430 0x7f67880913c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:17.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.904+0000 7f678d8ce640 1 --2- 172.21.15.60:0/2572198602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6788096eb0 0x7f678806fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:17.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.904+0000 7f678d0cd640 1 -- 172.21.15.60:0/2572198602 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67880d1430 msgr2=0x7f67880913c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:17.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.904+0000 7f678d0cd640 1 --2- 172.21.15.60:0/2572198602 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67880d1430 0x7f67880913c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:17.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.904+0000 7f678d0cd640 1 -- 172.21.15.60:0/2572198602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6788096eb0 msgr2=0x7f678806fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:17.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.904+0000 7f678d0cd640 1 --2- 172.21.15.60:0/2572198602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6788096eb0 0x7f678806fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:17.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.904+0000 7f678d0cd640 1 -- 172.21.15.60:0/2572198602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f678806ccb0 con 0x7f67880d1050 2024-05-09T05:01:17.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.904+0000 7f678d0cd640 1 --2- 172.21.15.60:0/2572198602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67880d1050 0x7f678806dd10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f677800c3c0 tx=0x7f677800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:17.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.904+0000 7f677e7fc640 1 -- 172.21.15.60:0/2572198602 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67780074e0 con 0x7f67880d1050 2024-05-09T05:01:17.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.904+0000 7f677e7fc640 1 -- 172.21.15.60:0/2572198602 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67780047e0 con 0x7f67880d1050 2024-05-09T05:01:17.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.904+0000 7f678fb32640 1 -- 172.21.15.60:0/2572198602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f678806cf50 con 0x7f67880d1050 2024-05-09T05:01:17.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.904+0000 7f677e7fc640 1 -- 172.21.15.60:0/2572198602 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6778004c10 con 0x7f67880d1050 2024-05-09T05:01:17.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.904+0000 7f678fb32640 1 -- 172.21.15.60:0/2572198602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f678806d7f0 con 0x7f67880d1050 2024-05-09T05:01:17.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.905+0000 7f678fb32640 1 -- 172.21.15.60:0/2572198602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f678806d120 con 0x7f67880d1050 2024-05-09T05:01:17.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.906+0000 7f677e7fc640 1 -- 172.21.15.60:0/2572198602 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6778011070 con 0x7f67880d1050 2024-05-09T05:01:17.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.906+0000 7f677e7fc640 1 --2- 172.21.15.60:0/2572198602 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f676c0762d0 0x7f676c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:17.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.907+0000 7f677e7fc640 1 -- 172.21.15.60:0/2572198602 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6464+0+0 (secure 0 0 0) 0x7f67780972b0 con 0x7f67880d1050 2024-05-09T05:01:17.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.907+0000 7f678d8ce640 1 --2- 172.21.15.60:0/2572198602 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f676c0762d0 0x7f676c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:17.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.907+0000 7f678d8ce640 1 --2- 172.21.15.60:0/2572198602 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f676c0762d0 0x7f676c078790 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f67800079a0 tx=0x7f6780008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:17.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:17.911+0000 7f677e7fc640 1 -- 172.21.15.60:0/2572198602 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6778060360 con 0x7f67880d1050 2024-05-09T05:01:18.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.142+0000 7f678fb32640 1 -- 172.21.15.60:0/2572198602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"} v 0) -- 0x7f6788075540 con 0x7f67880d1050 2024-05-09T05:01:18.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.766+0000 7f677e7fc640 1 -- 172.21.15.60:0/2572198602 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"}]=0 added data pool 7 to fsmap v18) ==== 131+0+0 (secure 0 0 0) 0x7f6778064660 con 0x7f67880d1050 2024-05-09T05:01:18.766 INFO:teuthology.orchestra.run.smithi060.stderr:added data pool 7 to fsmap 2024-05-09T05:01:18.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.768+0000 7f678fb32640 1 -- 172.21.15.60:0/2572198602 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f676c0762d0 msgr2=0x7f676c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:18.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.768+0000 7f678fb32640 1 --2- 172.21.15.60:0/2572198602 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f676c0762d0 0x7f676c078790 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f67800079a0 tx=0x7f6780008040 comp rx=0 tx=0).stop 2024-05-09T05:01:18.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.768+0000 7f678fb32640 1 -- 172.21.15.60:0/2572198602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67880d1050 msgr2=0x7f678806dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:18.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.768+0000 7f678fb32640 1 --2- 172.21.15.60:0/2572198602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67880d1050 0x7f678806dd10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f677800c3c0 tx=0x7f677800c880 comp rx=0 tx=0).stop 2024-05-09T05:01:18.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.769+0000 7f678fb32640 1 -- 172.21.15.60:0/2572198602 shutdown_connections 2024-05-09T05:01:18.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.769+0000 7f678fb32640 1 --2- 172.21.15.60:0/2572198602 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f676c0762d0 0x7f676c078790 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:18.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.769+0000 7f678fb32640 1 --2- 172.21.15.60:0/2572198602 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f67880d1430 0x7f67880913c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:18.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.769+0000 7f678fb32640 1 --2- 172.21.15.60:0/2572198602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67880d1050 0x7f678806dd10 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:18.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.769+0000 7f678fb32640 1 --2- 172.21.15.60:0/2572198602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6788096eb0 0x7f678806fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:18.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.769+0000 7f678fb32640 1 -- 172.21.15.60:0/2572198602 >> 172.21.15.60:0/2572198602 conn(0x7f67880993f0 msgr2=0x7f67880af590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:18.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.769+0000 7f678fb32640 1 -- 172.21.15.60:0/2572198602 shutdown_connections 2024-05-09T05:01:18.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.769+0000 7f678fb32640 1 -- 172.21.15.60:0/2572198602 wait complete. 2024-05-09T05:01:18.793 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:18.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.902+0000 7f48e035f640 1 Processor -- start 2024-05-09T05:01:18.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.902+0000 7f48e035f640 1 -- start start 2024-05-09T05:01:18.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.903+0000 7f48e035f640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f48d815f300 0x7f48d81639f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:18.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.903+0000 7f48e035f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48d81606f0 0x7f48d806ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:18.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.903+0000 7f48e035f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48d806ff60 con 0x7f48d806f9f0 2024-05-09T05:01:18.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.903+0000 7f48e035f640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f48d8070240 con 0x7f48d815f300 2024-05-09T05:01:18.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.903+0000 7f48e035f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f48d8070520 con 0x7f48d81606f0 2024-05-09T05:01:18.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.903+0000 7f48de8fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48d81606f0 0x7f48d806ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:18.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.903+0000 7f48de0fb640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f48d815f300 0x7f48d81639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:18.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.903+0000 7f48de8fc640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48d81606f0 0x7f48d806ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.60:34134/0 (socket says 172.21.15.60:34134) 2024-05-09T05:01:18.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.903+0000 7f48de8fc640 1 -- 172.21.15.60:0/2756899917 learned_addr learned my addr 172.21.15.60:0/2756899917 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:18.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.903+0000 7f48dd8fa640 1 --1- 172.21.15.60:0/2756899917 >> v1:172.21.15.114:6789/0 conn(0x7f48d806f9f0 0x7f48d80d33f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.60:41086/0 (socket says 172.21.15.60:41086) 2024-05-09T05:01:18.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.904+0000 7f48de8fc640 1 -- 172.21.15.60:0/2756899917 >> v2:172.21.15.114:3300/0 conn(0x7f48d815f300 msgr2=0x7f48d81639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:18.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.904+0000 7f48de8fc640 1 --2- 172.21.15.60:0/2756899917 >> v2:172.21.15.114:3300/0 conn(0x7f48d815f300 0x7f48d81639f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:18.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.904+0000 7f48de8fc640 1 -- 172.21.15.60:0/2756899917 >> v1:172.21.15.114:6789/0 conn(0x7f48d806f9f0 legacy=0x7f48d80d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:18.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.904+0000 7f48de8fc640 1 -- 172.21.15.60:0/2756899917 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d806e740 con 0x7f48d81606f0 2024-05-09T05:01:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.905+0000 7f48de8fc640 1 --2- 172.21.15.60:0/2756899917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48d81606f0 0x7f48d806ef40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f48d0009e50 tx=0x7f48d002a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b4cd6ced622b48d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.905+0000 7f48dd0f9640 1 -- 172.21.15.60:0/2756899917 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d0036070 con 0x7f48d81606f0 2024-05-09T05:01:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.905+0000 7f48dd0f9640 1 -- 172.21.15.60:0/2756899917 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48d0031600 con 0x7f48d81606f0 2024-05-09T05:01:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.905+0000 7f48dd0f9640 1 -- 172.21.15.60:0/2756899917 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d00319d0 con 0x7f48d81606f0 2024-05-09T05:01:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.905+0000 7f48e035f640 1 -- 172.21.15.60:0/2756899917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48d81606f0 msgr2=0x7f48d806ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.905+0000 7f48e035f640 1 --2- 172.21.15.60:0/2756899917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48d81606f0 0x7f48d806ef40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f48d0009e50 tx=0x7f48d002a710 comp rx=0 tx=0).stop 2024-05-09T05:01:18.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.905+0000 7f48e035f640 1 -- 172.21.15.60:0/2756899917 shutdown_connections 2024-05-09T05:01:18.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.905+0000 7f48e035f640 1 --2- 172.21.15.60:0/2756899917 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48d81606f0 0x7f48d806ef40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:18.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.905+0000 7f48e035f640 1 --2- 172.21.15.60:0/2756899917 >> v2:172.21.15.114:3300/0 conn(0x7f48d815f300 0x7f48d81639f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:18.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.905+0000 7f48e035f640 1 -- 172.21.15.60:0/2756899917 >> 172.21.15.60:0/2756899917 conn(0x7f48d809b850 msgr2=0x7f48d809bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:18.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.905+0000 7f48e035f640 1 -- 172.21.15.60:0/2756899917 shutdown_connections 2024-05-09T05:01:18.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.905+0000 7f48e035f640 1 -- 172.21.15.60:0/2756899917 wait complete. 2024-05-09T05:01:18.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.907+0000 7f48e035f640 1 Processor -- start 2024-05-09T05:01:18.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.907+0000 7f48e035f640 1 -- start start 2024-05-09T05:01:18.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.907+0000 7f48e035f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48d806f9f0 0x7f48d809ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:18.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.907+0000 7f48e035f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48d815f300 0x7f48d80b3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:18.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.907+0000 7f48e035f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48d81606f0 0x7f48d81f4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:18.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.907+0000 7f48e035f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f48d80d3b00 con 0x7f48d81606f0 2024-05-09T05:01:18.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.907+0000 7f48e035f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f48d8163f30 con 0x7f48d815f300 2024-05-09T05:01:18.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.907+0000 7f48e035f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f48d80d3c80 con 0x7f48d806f9f0 2024-05-09T05:01:18.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.908+0000 7f48de8fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48d81606f0 0x7f48d81f4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:18.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.908+0000 7f48de0fb640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48d806f9f0 0x7f48d809ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:18.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.908+0000 7f48de8fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48d81606f0 0x7f48d81f4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59336/0 (socket says 172.21.15.60:59336) 2024-05-09T05:01:18.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.908+0000 7f48de8fc640 1 -- 172.21.15.60:0/3720701870 learned_addr learned my addr 172.21.15.60:0/3720701870 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:18.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.908+0000 7f48dd8fa640 1 --2- 172.21.15.60:0/3720701870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48d815f300 0x7f48d80b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:18.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.908+0000 7f48de8fc640 1 -- 172.21.15.60:0/3720701870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48d806f9f0 msgr2=0x7f48d809ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:18.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.908+0000 7f48de8fc640 1 --2- 172.21.15.60:0/3720701870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48d806f9f0 0x7f48d809ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:18.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.908+0000 7f48de8fc640 1 -- 172.21.15.60:0/3720701870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48d815f300 msgr2=0x7f48d80b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:18.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.908+0000 7f48de8fc640 1 --2- 172.21.15.60:0/3720701870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48d815f300 0x7f48d80b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:18.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.908+0000 7f48de8fc640 1 -- 172.21.15.60:0/3720701870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d80b1e10 con 0x7f48d81606f0 2024-05-09T05:01:18.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.908+0000 7f48de0fb640 1 --2- 172.21.15.60:0/3720701870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48d806f9f0 0x7f48d809ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:01:18.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.908+0000 7f48de8fc640 1 --2- 172.21.15.60:0/3720701870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48d81606f0 0x7f48d81f4de0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f48d0034040 tx=0x7f48d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:18.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.908+0000 7f48ceffd640 1 -- 172.21.15.60:0/3720701870 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d0041020 con 0x7f48d81606f0 2024-05-09T05:01:18.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.908+0000 7f48ceffd640 1 -- 172.21.15.60:0/3720701870 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48d0007820 con 0x7f48d81606f0 2024-05-09T05:01:18.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.908+0000 7f48ceffd640 1 -- 172.21.15.60:0/3720701870 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d0036040 con 0x7f48d81606f0 2024-05-09T05:01:18.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.908+0000 7f48e035f640 1 -- 172.21.15.60:0/3720701870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48d80b0dd0 con 0x7f48d81606f0 2024-05-09T05:01:18.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.908+0000 7f48e035f640 1 -- 172.21.15.60:0/3720701870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48d80b1610 con 0x7f48d81606f0 2024-05-09T05:01:18.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.909+0000 7f48e035f640 1 -- 172.21.15.60:0/3720701870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48d8051040 con 0x7f48d81606f0 2024-05-09T05:01:18.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.910+0000 7f48ceffd640 1 -- 172.21.15.60:0/3720701870 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f48d0044020 con 0x7f48d81606f0 2024-05-09T05:01:18.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.911+0000 7f48ceffd640 1 --2- 172.21.15.60:0/3720701870 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f48bc0762d0 0x7f48bc078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:18.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.911+0000 7f48ceffd640 1 -- 172.21.15.60:0/3720701870 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6496+0+0 (secure 0 0 0) 0x7f48d00b8350 con 0x7f48d81606f0 2024-05-09T05:01:18.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.911+0000 7f48de0fb640 1 --2- 172.21.15.60:0/3720701870 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f48bc0762d0 0x7f48bc078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:18.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.911+0000 7f48de0fb640 1 --2- 172.21.15.60:0/3720701870 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f48bc0762d0 0x7f48bc078790 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009f30 tx=0x7f48d4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:18.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:18.915+0000 7f48ceffd640 1 -- 172.21.15.60:0/3720701870 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f48d004b030 con 0x7f48d81606f0 2024-05-09T05:01:19.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.147+0000 7f48e035f640 1 -- 172.21.15.60:0/3720701870 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48d809f8e0 con 0x7f48d81606f0 2024-05-09T05:01:19.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.148+0000 7f48ceffd640 1 -- 172.21.15.60:0/3720701870 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5183 (secure 0 0 0) 0x7f48d80b0dd0 con 0x7f48d81606f0 2024-05-09T05:01:19.149 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:19.149 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4547,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2372252858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2372252858},{"type":"v1","addr":"172.21.15.114:6836","nonce":2372252858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1761282610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1761282610},{"type":"v1","addr":"172.21.15.114:6837","nonce":1761282610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4765,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/300408012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":300408012},{"type":"v1","addr":"172.21.15.60:6835","nonce":300408012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:04.698975+0000","modified":"2024-05-09T05:01:18.144520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4893},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4893":{"gid":4893,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/399160914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":399160914},{"type":"v1","addr":"172.21.15.60:6834","nonce":399160914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4893,"qdb_cluster":[4893]},"id":2}]} 2024-05-09T05:01:19.149 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 18 2024-05-09T05:01:19.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.150+0000 7f48e035f640 1 -- 172.21.15.60:0/3720701870 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f48bc0762d0 msgr2=0x7f48bc078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:19.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.150+0000 7f48e035f640 1 --2- 172.21.15.60:0/3720701870 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f48bc0762d0 0x7f48bc078790 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009f30 tx=0x7f48d4009190 comp rx=0 tx=0).stop 2024-05-09T05:01:19.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.150+0000 7f48e035f640 1 -- 172.21.15.60:0/3720701870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48d81606f0 msgr2=0x7f48d81f4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:19.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.150+0000 7f48e035f640 1 --2- 172.21.15.60:0/3720701870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48d81606f0 0x7f48d81f4de0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f48d0034040 tx=0x7f48d0004a40 comp rx=0 tx=0).stop 2024-05-09T05:01:19.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.150+0000 7f48e035f640 1 -- 172.21.15.60:0/3720701870 shutdown_connections 2024-05-09T05:01:19.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.150+0000 7f48e035f640 1 --2- 172.21.15.60:0/3720701870 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f48bc0762d0 0x7f48bc078790 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.150+0000 7f48e035f640 1 --2- 172.21.15.60:0/3720701870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48d81606f0 0x7f48d81f4de0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.150+0000 7f48e035f640 1 --2- 172.21.15.60:0/3720701870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48d815f300 0x7f48d80b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.150+0000 7f48e035f640 1 --2- 172.21.15.60:0/3720701870 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f48d806f9f0 0x7f48d809ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.150+0000 7f48e035f640 1 -- 172.21.15.60:0/3720701870 >> 172.21.15.60:0/3720701870 conn(0x7f48d809b850 msgr2=0x7f48d8162ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:19.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.150+0000 7f48e035f640 1 -- 172.21.15.60:0/3720701870 shutdown_connections 2024-05-09T05:01:19.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.150+0000 7f48e035f640 1 -- 172.21.15.60:0/3720701870 wait complete. 2024-05-09T05:01:19.161 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:01:19.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.265+0000 7fc63f746640 1 Processor -- start 2024-05-09T05:01:19.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.265+0000 7fc63f746640 1 -- start start 2024-05-09T05:01:19.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.266+0000 7fc63f746640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc6380ae1f0 0x7fc63809c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:19.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.266+0000 7fc63f746640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc63809cb40 0x7fc6380b1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:19.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.266+0000 7fc63f746640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6380b3d40 con 0x7fc6380b1580 2024-05-09T05:01:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.266+0000 7fc63f746640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fc6380b28a0 con 0x7fc6380ae1f0 2024-05-09T05:01:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.266+0000 7fc63f746640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fc63809d890 con 0x7fc63809cb40 2024-05-09T05:01:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.266+0000 7fc63d4e2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc6380ae1f0 0x7fc63809c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.266+0000 7fc63d4e2640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc6380ae1f0 0x7fc63809c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59346/0 (socket says 172.21.15.60:59346) 2024-05-09T05:01:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.266+0000 7fc63d4e2640 1 -- 172.21.15.60:0/161807429 learned_addr learned my addr 172.21.15.60:0/161807429 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.267+0000 7fc63cce1640 1 --2- 172.21.15.60:0/161807429 >> v2:172.21.15.114:3300/0 conn(0x7fc63809cb40 0x7fc6380b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.267+0000 7fc63d4e2640 1 -- 172.21.15.60:0/161807429 >> v2:172.21.15.114:3300/0 conn(0x7fc63809cb40 msgr2=0x7fc6380b1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.267+0000 7fc63d4e2640 1 --2- 172.21.15.60:0/161807429 >> v2:172.21.15.114:3300/0 conn(0x7fc63809cb40 0x7fc6380b1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.267+0000 7fc63d4e2640 1 -- 172.21.15.60:0/161807429 >> v1:172.21.15.114:6789/0 conn(0x7fc6380b1580 legacy=0x7fc6380aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.267+0000 7fc63d4e2640 1 -- 172.21.15.60:0/161807429 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63809d2d0 con 0x7fc6380ae1f0 2024-05-09T05:01:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.267+0000 7fc63cce1640 1 --2- 172.21.15.60:0/161807429 >> v2:172.21.15.114:3300/0 conn(0x7fc63809cb40 0x7fc6380b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:19.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.268+0000 7fc63d4e2640 1 --2- 172.21.15.60:0/161807429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6380ae1f0 0x7fc63809c600 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc63809abb0 tx=0x7fc62802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7366273b219c46ab server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:19.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.268+0000 7fc637fff640 1 -- 172.21.15.60:0/161807429 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc628036070 con 0x7fc6380ae1f0 2024-05-09T05:01:19.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.268+0000 7fc637fff640 1 -- 172.21.15.60:0/161807429 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc628031600 con 0x7fc6380ae1f0 2024-05-09T05:01:19.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.268+0000 7fc637fff640 1 -- 172.21.15.60:0/161807429 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6280319d0 con 0x7fc6380ae1f0 2024-05-09T05:01:19.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.268+0000 7fc63f746640 1 -- 172.21.15.60:0/161807429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6380ae1f0 msgr2=0x7fc63809c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:19.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.268+0000 7fc63f746640 1 --2- 172.21.15.60:0/161807429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6380ae1f0 0x7fc63809c600 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc63809abb0 tx=0x7fc62802a730 comp rx=0 tx=0).stop 2024-05-09T05:01:19.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.268+0000 7fc63f746640 1 -- 172.21.15.60:0/161807429 shutdown_connections 2024-05-09T05:01:19.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.268+0000 7fc63f746640 1 --2- 172.21.15.60:0/161807429 >> v2:172.21.15.114:3300/0 conn(0x7fc63809cb40 0x7fc6380b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.268+0000 7fc63f746640 1 --2- 172.21.15.60:0/161807429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6380ae1f0 0x7fc63809c600 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.268+0000 7fc63f746640 1 -- 172.21.15.60:0/161807429 >> 172.21.15.60:0/161807429 conn(0x7fc638072390 msgr2=0x7fc6380727a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:19.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.268+0000 7fc63f746640 1 -- 172.21.15.60:0/161807429 shutdown_connections 2024-05-09T05:01:19.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.268+0000 7fc63f746640 1 -- 172.21.15.60:0/161807429 wait complete. 2024-05-09T05:01:19.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.269+0000 7fc63f746640 1 Processor -- start 2024-05-09T05:01:19.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.270+0000 7fc63f746640 1 -- start start 2024-05-09T05:01:19.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.270+0000 7fc63f746640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc63809cb40 0x7fc638159e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:19.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.270+0000 7fc63f746640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6380ae1f0 0x7fc638161760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:19.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.270+0000 7fc63f746640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6380b1580 0x7fc638161ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:19.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.270+0000 7fc63f746640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc63809b720 con 0x7fc6380ae1f0 2024-05-09T05:01:19.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.270+0000 7fc63f746640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc6380b2530 con 0x7fc63809cb40 2024-05-09T05:01:19.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.270+0000 7fc63f746640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc6380b19f0 con 0x7fc6380b1580 2024-05-09T05:01:19.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.270+0000 7fc63cce1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6380ae1f0 0x7fc638161760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:19.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.270+0000 7fc63cce1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6380ae1f0 0x7fc638161760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59358/0 (socket says 172.21.15.60:59358) 2024-05-09T05:01:19.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.270+0000 7fc63cce1640 1 -- 172.21.15.60:0/2028556989 learned_addr learned my addr 172.21.15.60:0/2028556989 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:19.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.271+0000 7fc63dce3640 1 --2- 172.21.15.60:0/2028556989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6380b1580 0x7fc638161ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:19.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.271+0000 7fc63d4e2640 1 --2- 172.21.15.60:0/2028556989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc63809cb40 0x7fc638159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:19.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.271+0000 7fc63cce1640 1 -- 172.21.15.60:0/2028556989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6380b1580 msgr2=0x7fc638161ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:19.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.271+0000 7fc63cce1640 1 --2- 172.21.15.60:0/2028556989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6380b1580 0x7fc638161ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.271+0000 7fc63cce1640 1 -- 172.21.15.60:0/2028556989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc63809cb40 msgr2=0x7fc638159e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:19.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.271+0000 7fc63cce1640 1 --2- 172.21.15.60:0/2028556989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc63809cb40 0x7fc638159e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.271+0000 7fc63cce1640 1 -- 172.21.15.60:0/2028556989 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63815a350 con 0x7fc6380ae1f0 2024-05-09T05:01:19.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.271+0000 7fc63dce3640 1 --2- 172.21.15.60:0/2028556989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6380b1580 0x7fc638161ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:19.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.271+0000 7fc63cce1640 1 --2- 172.21.15.60:0/2028556989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6380ae1f0 0x7fc638161760 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00e810 tx=0x7fc62c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:19.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.271+0000 7fc635ffb640 1 -- 172.21.15.60:0/2028556989 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c00f040 con 0x7fc6380ae1f0 2024-05-09T05:01:19.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.271+0000 7fc635ffb640 1 -- 172.21.15.60:0/2028556989 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc62c004030 con 0x7fc6380ae1f0 2024-05-09T05:01:19.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.271+0000 7fc63f746640 1 -- 172.21.15.60:0/2028556989 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc63815a710 con 0x7fc6380ae1f0 2024-05-09T05:01:19.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.271+0000 7fc635ffb640 1 -- 172.21.15.60:0/2028556989 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c004400 con 0x7fc6380ae1f0 2024-05-09T05:01:19.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.271+0000 7fc63f746640 1 -- 172.21.15.60:0/2028556989 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc63815acd0 con 0x7fc6380ae1f0 2024-05-09T05:01:19.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.272+0000 7fc63f746640 1 -- 172.21.15.60:0/2028556989 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc638051040 con 0x7fc6380ae1f0 2024-05-09T05:01:19.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.273+0000 7fc635ffb640 1 -- 172.21.15.60:0/2028556989 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc62c00c6d0 con 0x7fc6380ae1f0 2024-05-09T05:01:19.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.274+0000 7fc635ffb640 1 --2- 172.21.15.60:0/2028556989 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc624076300 0x7fc6240787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:19.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.274+0000 7fc635ffb640 1 -- 172.21.15.60:0/2028556989 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6496+0+0 (secure 0 0 0) 0x7fc62c0975d0 con 0x7fc6380ae1f0 2024-05-09T05:01:19.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.274+0000 7fc63d4e2640 1 --2- 172.21.15.60:0/2028556989 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc624076300 0x7fc6240787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:19.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.274+0000 7fc63d4e2640 1 --2- 172.21.15.60:0/2028556989 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc624076300 0x7fc6240787c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc628033000 tx=0x7fc628009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:19.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.278+0000 7fc635ffb640 1 -- 172.21.15.60:0/2028556989 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc62c017020 con 0x7fc6380ae1f0 2024-05-09T05:01:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.471+0000 7fc63f746640 1 -- 172.21.15.60:0/2028556989 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc638096e30 con 0x7fc6380ae1f0 2024-05-09T05:01:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.473+0000 7fc635ffb640 1 -- 172.21.15.60:0/2028556989 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+18743 (secure 0 0 0) 0x7fc62c060cb0 con 0x7fc6380ae1f0 2024-05-09T05:01:19.474 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:19.474 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":38,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:01:18.144315+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:01:02.773384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-05-09T05:01:03.658182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-05-09T05:01:16.847013+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:01:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.475+0000 7fc63f746640 1 -- 172.21.15.60:0/2028556989 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc624076300 msgr2=0x7fc6240787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.475+0000 7fc63f746640 1 --2- 172.21.15.60:0/2028556989 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc624076300 0x7fc6240787c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc628033000 tx=0x7fc628009000 comp rx=0 tx=0).stop 2024-05-09T05:01:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.475+0000 7fc63f746640 1 -- 172.21.15.60:0/2028556989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6380ae1f0 msgr2=0x7fc638161760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.475+0000 7fc63f746640 1 --2- 172.21.15.60:0/2028556989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6380ae1f0 0x7fc638161760 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00e810 tx=0x7fc62c00ecd0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.475+0000 7fc63f746640 1 -- 172.21.15.60:0/2028556989 shutdown_connections 2024-05-09T05:01:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.475+0000 7fc63f746640 1 --2- 172.21.15.60:0/2028556989 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc624076300 0x7fc6240787c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.475+0000 7fc63f746640 1 --2- 172.21.15.60:0/2028556989 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc6380b1580 0x7fc638161ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.475+0000 7fc63f746640 1 --2- 172.21.15.60:0/2028556989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6380ae1f0 0x7fc638161760 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.475+0000 7fc63f746640 1 --2- 172.21.15.60:0/2028556989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc63809cb40 0x7fc638159e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.475+0000 7fc63f746640 1 -- 172.21.15.60:0/2028556989 >> 172.21.15.60:0/2028556989 conn(0x7fc638072390 msgr2=0x7fc63809bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.475+0000 7fc63f746640 1 -- 172.21.15.60:0/2028556989 shutdown_connections 2024-05-09T05:01:19.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.475+0000 7fc63f746640 1 -- 172.21.15.60:0/2028556989 wait complete. 2024-05-09T05:01:19.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:19.487 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.pool -v data_new ./parent_b/alpha) 2024-05-09T05:01:19.507 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:19.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.651+0000 7f5163d22640 1 Processor -- start 2024-05-09T05:01:19.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.652+0000 7f5163d22640 1 -- start start 2024-05-09T05:01:19.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.652+0000 7f5163d22640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f515c1606f0 0x7f515c0d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:19.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.652+0000 7f5163d22640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f515c06f9f0 0x7f515c0d3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:19.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.652+0000 7f5163d22640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f515c06ff60 con 0x7f515c15f300 2024-05-09T05:01:19.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.652+0000 7f5163d22640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f515c070240 con 0x7f515c06f9f0 2024-05-09T05:01:19.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.652+0000 7f5163d22640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f515c070520 con 0x7f515c1606f0 2024-05-09T05:01:19.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.652+0000 7f51622bf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f515c06f9f0 0x7f515c0d3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:19.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.653+0000 7f51622bf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f515c06f9f0 0x7f515c0d3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59370/0 (socket says 172.21.15.60:59370) 2024-05-09T05:01:19.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.653+0000 7f51622bf640 1 -- 172.21.15.60:0/929375360 learned_addr learned my addr 172.21.15.60:0/929375360 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:19.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.653+0000 7f51612bd640 1 --2- 172.21.15.60:0/929375360 >> v2:172.21.15.114:3300/0 conn(0x7f515c1606f0 0x7f515c0d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:19.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.653+0000 7f51622bf640 1 -- 172.21.15.60:0/929375360 >> v2:172.21.15.114:3300/0 conn(0x7f515c1606f0 msgr2=0x7f515c0d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:19.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.653+0000 7f51622bf640 1 --2- 172.21.15.60:0/929375360 >> v2:172.21.15.114:3300/0 conn(0x7f515c1606f0 0x7f515c0d33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.653+0000 7f51622bf640 1 -- 172.21.15.60:0/929375360 >> v1:172.21.15.114:6789/0 conn(0x7f515c15f300 legacy=0x7f515c1639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:19.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.653+0000 7f51622bf640 1 -- 172.21.15.60:0/929375360 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f515c06e740 con 0x7f515c06f9f0 2024-05-09T05:01:19.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.654+0000 7f51622bf640 1 --2- 172.21.15.60:0/929375360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f515c06f9f0 0x7f515c0d3930 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f514c009e50 tx=0x7f514c02a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dd20343b66234e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:19.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.654+0000 7f5160abc640 1 -- 172.21.15.60:0/929375360 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c036070 con 0x7f515c06f9f0 2024-05-09T05:01:19.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.654+0000 7f5160abc640 1 -- 172.21.15.60:0/929375360 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f514c031600 con 0x7f515c06f9f0 2024-05-09T05:01:19.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.654+0000 7f5160abc640 1 -- 172.21.15.60:0/929375360 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c0319d0 con 0x7f515c06f9f0 2024-05-09T05:01:19.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.654+0000 7f5163d22640 1 -- 172.21.15.60:0/929375360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f515c06f9f0 msgr2=0x7f515c0d3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:19.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.654+0000 7f5163d22640 1 --2- 172.21.15.60:0/929375360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f515c06f9f0 0x7f515c0d3930 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f514c009e50 tx=0x7f514c02a760 comp rx=0 tx=0).stop 2024-05-09T05:01:19.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.654+0000 7f5163d22640 1 -- 172.21.15.60:0/929375360 shutdown_connections 2024-05-09T05:01:19.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.654+0000 7f5163d22640 1 --2- 172.21.15.60:0/929375360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f515c06f9f0 0x7f515c0d3930 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.654+0000 7f5163d22640 1 --2- 172.21.15.60:0/929375360 >> v2:172.21.15.114:3300/0 conn(0x7f515c1606f0 0x7f515c0d33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.654+0000 7f5163d22640 1 -- 172.21.15.60:0/929375360 >> 172.21.15.60:0/929375360 conn(0x7f515c09b850 msgr2=0x7f515c09bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:19.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.654+0000 7f5163d22640 1 -- 172.21.15.60:0/929375360 shutdown_connections 2024-05-09T05:01:19.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.654+0000 7f5163d22640 1 -- 172.21.15.60:0/929375360 wait complete. 2024-05-09T05:01:19.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.656+0000 7f5163d22640 1 Processor -- start 2024-05-09T05:01:19.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.656+0000 7f5163d22640 1 -- start start 2024-05-09T05:01:19.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.656+0000 7f5163d22640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f515c06f9f0 0x7f515c1f6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:19.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.656+0000 7f5163d22640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f515c15f300 0x7f515c1611c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:19.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.656+0000 7f5163d22640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f515c1606f0 0x7f515c150080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:19.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.656+0000 7f5163d22640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f515c06f0c0 con 0x7f515c1606f0 2024-05-09T05:01:19.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.656+0000 7f5163d22640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f515c06ef40 con 0x7f515c15f300 2024-05-09T05:01:19.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.656+0000 7f5163d22640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f515c06f240 con 0x7f515c06f9f0 2024-05-09T05:01:19.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.656+0000 7f51622bf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f515c1606f0 0x7f515c150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:19.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.657+0000 7f51622bf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f515c1606f0 0x7f515c150080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59382/0 (socket says 172.21.15.60:59382) 2024-05-09T05:01:19.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.657+0000 7f51622bf640 1 -- 172.21.15.60:0/4292487308 learned_addr learned my addr 172.21.15.60:0/4292487308 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:19.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.657+0000 7f51612bd640 1 --2- 172.21.15.60:0/4292487308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f515c15f300 0x7f515c1611c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:19.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.657+0000 7f5161abe640 1 --2- 172.21.15.60:0/4292487308 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f515c06f9f0 0x7f515c1f6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:19.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.657+0000 7f51622bf640 1 -- 172.21.15.60:0/4292487308 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f515c06f9f0 msgr2=0x7f515c1f6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:19.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.657+0000 7f51622bf640 1 --2- 172.21.15.60:0/4292487308 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f515c06f9f0 0x7f515c1f6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.657+0000 7f51622bf640 1 -- 172.21.15.60:0/4292487308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f515c15f300 msgr2=0x7f515c1611c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:19.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.657+0000 7f51622bf640 1 --2- 172.21.15.60:0/4292487308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f515c15f300 0x7f515c1611c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.657+0000 7f51622bf640 1 -- 172.21.15.60:0/4292487308 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f515c1f9400 con 0x7f515c1606f0 2024-05-09T05:01:19.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.657+0000 7f5161abe640 1 --2- 172.21.15.60:0/4292487308 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f515c06f9f0 0x7f515c1f6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:19.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.657+0000 7f51622bf640 1 --2- 172.21.15.60:0/4292487308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f515c1606f0 0x7f515c150080 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f514c034040 tx=0x7f514c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:19.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.657+0000 7f515a7fc640 1 -- 172.21.15.60:0/4292487308 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c036050 con 0x7f515c1606f0 2024-05-09T05:01:19.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.657+0000 7f515a7fc640 1 -- 172.21.15.60:0/4292487308 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f514c042070 con 0x7f515c1606f0 2024-05-09T05:01:19.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.657+0000 7f515a7fc640 1 -- 172.21.15.60:0/4292487308 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c009510 con 0x7f515c1606f0 2024-05-09T05:01:19.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.657+0000 7f5163d22640 1 -- 172.21.15.60:0/4292487308 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f515c1f96e0 con 0x7f515c1606f0 2024-05-09T05:01:19.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.657+0000 7f5163d22640 1 -- 172.21.15.60:0/4292487308 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f515c1f9ca0 con 0x7f515c1606f0 2024-05-09T05:01:19.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.658+0000 7f5163d22640 1 -- 172.21.15.60:0/4292487308 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f515c0d0970 con 0x7f515c1606f0 2024-05-09T05:01:19.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.659+0000 7f515a7fc640 1 -- 172.21.15.60:0/4292487308 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f514c007bf0 con 0x7f515c1606f0 2024-05-09T05:01:19.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.659+0000 7f515a7fc640 1 --2- 172.21.15.60:0/4292487308 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5148076300 0x7f51480787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:19.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.659+0000 7f515a7fc640 1 -- 172.21.15.60:0/4292487308 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6496+0+0 (secure 0 0 0) 0x7f514c0b85a0 con 0x7f515c1606f0 2024-05-09T05:01:19.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.659+0000 7f5161abe640 1 --2- 172.21.15.60:0/4292487308 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5148076300 0x7f51480787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:19.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.660+0000 7f5161abe640 1 --2- 172.21.15.60:0/4292487308 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5148076300 0x7f51480787c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5154001740 tx=0x7f5154009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:19.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.664+0000 7f515a7fc640 1 -- 172.21.15.60:0/4292487308 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f515c0d0970 con 0x7f515c1606f0 2024-05-09T05:01:19.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.900+0000 7f5163d22640 1 -- 172.21.15.60:0/4292487308 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f515c051040 con 0x7f515c1606f0 2024-05-09T05:01:19.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.900+0000 7f515a7fc640 1 -- 172.21.15.60:0/4292487308 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5183 (secure 0 0 0) 0x7f514c081bf0 con 0x7f515c1606f0 2024-05-09T05:01:19.901 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:19.901 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4547,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2372252858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2372252858},{"type":"v1","addr":"172.21.15.114:6836","nonce":2372252858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1761282610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1761282610},{"type":"v1","addr":"172.21.15.114:6837","nonce":1761282610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4765,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/300408012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":300408012},{"type":"v1","addr":"172.21.15.60:6835","nonce":300408012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:04.698975+0000","modified":"2024-05-09T05:01:18.144520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4893},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4893":{"gid":4893,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/399160914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":399160914},{"type":"v1","addr":"172.21.15.60:6834","nonce":399160914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4893,"qdb_cluster":[4893]},"id":2}]} 2024-05-09T05:01:19.901 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 18 2024-05-09T05:01:19.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.903+0000 7f5163d22640 1 -- 172.21.15.60:0/4292487308 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5148076300 msgr2=0x7f51480787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:19.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.903+0000 7f5163d22640 1 --2- 172.21.15.60:0/4292487308 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5148076300 0x7f51480787c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5154001740 tx=0x7f5154009040 comp rx=0 tx=0).stop 2024-05-09T05:01:19.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.903+0000 7f5163d22640 1 -- 172.21.15.60:0/4292487308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f515c1606f0 msgr2=0x7f515c150080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:19.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.903+0000 7f5163d22640 1 --2- 172.21.15.60:0/4292487308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f515c1606f0 0x7f515c150080 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f514c034040 tx=0x7f514c004a40 comp rx=0 tx=0).stop 2024-05-09T05:01:19.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.903+0000 7f5163d22640 1 -- 172.21.15.60:0/4292487308 shutdown_connections 2024-05-09T05:01:19.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.903+0000 7f5163d22640 1 --2- 172.21.15.60:0/4292487308 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f5148076300 0x7f51480787c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.903+0000 7f5163d22640 1 --2- 172.21.15.60:0/4292487308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f515c1606f0 0x7f515c150080 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.903+0000 7f5163d22640 1 --2- 172.21.15.60:0/4292487308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f515c15f300 0x7f515c1611c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.903+0000 7f5163d22640 1 --2- 172.21.15.60:0/4292487308 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f515c06f9f0 0x7f515c1f6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:19.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.903+0000 7f5163d22640 1 -- 172.21.15.60:0/4292487308 >> 172.21.15.60:0/4292487308 conn(0x7f515c09b850 msgr2=0x7f515c099640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:19.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.903+0000 7f5163d22640 1 -- 172.21.15.60:0/4292487308 shutdown_connections 2024-05-09T05:01:19.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:19.903+0000 7f5163d22640 1 -- 172.21.15.60:0/4292487308 wait complete. 2024-05-09T05:01:19.914 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-05-09T05:01:20.047 INFO:teuthology.orchestra.run.smithi060.stdout:{"message":"","return_code":0} 2024-05-09T05:01:20.047 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-05-09T05:01:20.048 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001fb.00000000 parent 2024-05-09T05:01:20.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.092+0000 7f513b13ef40 1 Processor -- start 2024-05-09T05:01:20.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.092+0000 7f513b13ef40 1 -- start start 2024-05-09T05:01:20.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f513b13ef40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5128002940 0x5556f762e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f513b13ef40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5556f763c3d0 0x5556f756b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f513b13ef40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5556f756be80 0x5556f763aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f513b13ef40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5556f7511f50 con 0x7f5128002940 2024-05-09T05:01:20.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f513b13ef40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5556f7515f90 con 0x5556f763c3d0 2024-05-09T05:01:20.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f513b13ef40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x5556f7640470 con 0x5556f756be80 2024-05-09T05:01:20.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f5138c38640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5128002940 0x5556f762e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f5138c38640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5128002940 0x5556f762e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59410/0 (socket says 172.21.15.60:59410) 2024-05-09T05:01:20.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f5138c38640 1 -- 172.21.15.60:0/1770933828 learned_addr learned my addr 172.21.15.60:0/1770933828 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:20.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f5133fff640 1 --2- 172.21.15.60:0/1770933828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5556f763c3d0 0x5556f756b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f5139439640 1 --2- 172.21.15.60:0/1770933828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5556f756be80 0x5556f763aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f5138c38640 1 -- 172.21.15.60:0/1770933828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5556f756be80 msgr2=0x5556f763aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:20.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f5138c38640 1 --2- 172.21.15.60:0/1770933828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5556f756be80 0x5556f763aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f5138c38640 1 -- 172.21.15.60:0/1770933828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5556f763c3d0 msgr2=0x5556f756b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:20.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f5138c38640 1 --2- 172.21.15.60:0/1770933828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5556f763c3d0 0x5556f756b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f5138c38640 1 -- 172.21.15.60:0/1770933828 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556f76344f0 con 0x7f5128002940 2024-05-09T05:01:20.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f5138c38640 1 --2- 172.21.15.60:0/1770933828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5128002940 0x5556f762e660 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f512c000fb0 tx=0x7f512c001820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5ed569529eca195 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:20.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f5132ffd640 1 -- 172.21.15.60:0/1770933828 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c00e8e0 con 0x7f5128002940 2024-05-09T05:01:20.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f5132ffd640 1 -- 172.21.15.60:0/1770933828 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f512c013030 con 0x7f5128002940 2024-05-09T05:01:20.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.093+0000 7f5132ffd640 1 -- 172.21.15.60:0/1770933828 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c00ece0 con 0x7f5128002940 2024-05-09T05:01:20.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.094+0000 7f513b13ef40 1 -- 172.21.15.60:0/1770933828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5128002940 msgr2=0x5556f762e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.094+0000 7f513b13ef40 1 --2- 172.21.15.60:0/1770933828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5128002940 0x5556f762e660 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f512c000fb0 tx=0x7f512c001820 comp rx=0 tx=0).stop 2024-05-09T05:01:20.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.094+0000 7f513b13ef40 1 -- 172.21.15.60:0/1770933828 shutdown_connections 2024-05-09T05:01:20.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.094+0000 7f513b13ef40 1 --2- 172.21.15.60:0/1770933828 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5556f756be80 0x5556f763aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.094+0000 7f513b13ef40 1 --2- 172.21.15.60:0/1770933828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5556f763c3d0 0x5556f756b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.094+0000 7f513b13ef40 1 --2- 172.21.15.60:0/1770933828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5128002940 0x5556f762e660 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.094+0000 7f513b13ef40 1 -- 172.21.15.60:0/1770933828 >> 172.21.15.60:0/1770933828 conn(0x5556f75948d0 msgr2=0x5556f756b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:20.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.094+0000 7f513b13ef40 1 -- 172.21.15.60:0/1770933828 shutdown_connections 2024-05-09T05:01:20.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.094+0000 7f513b13ef40 1 -- 172.21.15.60:0/1770933828 wait complete. 2024-05-09T05:01:20.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.094+0000 7f513b13ef40 1 Processor -- start 2024-05-09T05:01:20.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.094+0000 7f513b13ef40 1 -- start start 2024-05-09T05:01:20.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.095+0000 7f513b13ef40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5556f756be80 0x5556f7630f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.095+0000 7f513b13ef40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5556f763c3d0 0x5556f76245d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.095+0000 7f513b13ef40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5128002940 0x5556f7624b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.095+0000 7f513b13ef40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x5556f762efa0 con 0x5556f756be80 2024-05-09T05:01:20.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.095+0000 7f513b13ef40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5556f7634be0 con 0x7f5128002940 2024-05-09T05:01:20.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.095+0000 7f513b13ef40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x5556f756d1c0 con 0x5556f763c3d0 2024-05-09T05:01:20.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.095+0000 7f5138c38640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5556f756be80 0x5556f7630f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.095+0000 7f5138c38640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5556f756be80 0x5556f7630f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59422/0 (socket says 172.21.15.60:59422) 2024-05-09T05:01:20.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.095+0000 7f5138c38640 1 -- 172.21.15.60:0/816827644 learned_addr learned my addr 172.21.15.60:0/816827644 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:20.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.095+0000 7f5133fff640 1 --2- 172.21.15.60:0/816827644 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5556f763c3d0 0x5556f76245d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.095+0000 7f5139439640 1 --2- 172.21.15.60:0/816827644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5128002940 0x5556f7624b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.095+0000 7f5138c38640 1 -- 172.21.15.60:0/816827644 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5556f763c3d0 msgr2=0x5556f76245d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.095+0000 7f5138c38640 1 --2- 172.21.15.60:0/816827644 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x5556f763c3d0 0x5556f76245d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.095+0000 7f5138c38640 1 -- 172.21.15.60:0/816827644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5128002940 msgr2=0x5556f7624b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.095+0000 7f5138c38640 1 --2- 172.21.15.60:0/816827644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5128002940 0x5556f7624b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.095+0000 7f5138c38640 1 -- 172.21.15.60:0/816827644 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556f7628a10 con 0x5556f756be80 2024-05-09T05:01:20.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.096+0000 7f5138c38640 1 --2- 172.21.15.60:0/816827644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5556f756be80 0x5556f7630f40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f512c001eb0 tx=0x7f512c001a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:20.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.096+0000 7f5131ffb640 1 -- 172.21.15.60:0/816827644 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c00f860 con 0x5556f756be80 2024-05-09T05:01:20.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.096+0000 7f5131ffb640 1 -- 172.21.15.60:0/816827644 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f512c00fa20 con 0x5556f756be80 2024-05-09T05:01:20.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.096+0000 7f513b13ef40 1 -- 172.21.15.60:0/816827644 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5556f7628770 con 0x5556f756be80 2024-05-09T05:01:20.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.096+0000 7f5131ffb640 1 -- 172.21.15.60:0/816827644 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c00fe10 con 0x5556f756be80 2024-05-09T05:01:20.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.096+0000 7f513b13ef40 1 -- 172.21.15.60:0/816827644 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556f7633d90 con 0x5556f756be80 2024-05-09T05:01:20.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.097+0000 7f5131ffb640 1 -- 172.21.15.60:0/816827644 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f512c019430 con 0x5556f756be80 2024-05-09T05:01:20.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.098+0000 7f5131ffb640 1 --2- 172.21.15.60:0/816827644 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f511c074ce0 0x7f511c0771a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.098+0000 7f5131ffb640 1 -- 172.21.15.60:0/816827644 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6496+0+0 (secure 0 0 0) 0x7f512c0985e0 con 0x5556f756be80 2024-05-09T05:01:20.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.098+0000 7f5133fff640 1 --2- 172.21.15.60:0/816827644 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f511c074ce0 0x7f511c0771a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.098+0000 7f513b13ef40 1 --2- 172.21.15.60:0/816827644 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x5556f7631480 0x5556f762d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.098+0000 7f513b13ef40 1 -- 172.21.15.60:0/816827644 --> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] -- osd_op(unknown.0.0:1 6.13 6:ca8fdbba:::100000001fb.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x5556f75685c0 con 0x5556f7631480 2024-05-09T05:01:20.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.098+0000 7f5139439640 1 --2- 172.21.15.60:0/816827644 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x5556f7631480 0x5556f762d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.098+0000 7f5133fff640 1 --2- 172.21.15.60:0/816827644 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f511c074ce0 0x7f511c0771a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5124056000 tx=0x7f5124051000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:20.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.099+0000 7f5139439640 1 --2- 172.21.15.60:0/816827644 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x5556f7631480 0x5556f762d0e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:20.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.099+0000 7f5139439640 1 -- 172.21.15.60:0/816827644 <== osd.3 v2:172.21.15.60:6824/1069806621 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (103) out=103b] v0'0 uv3 ondisk = 0) ==== 164+0+103 (crc 0 0 0) 0x7f51340015b0 con 0x5556f7631480 2024-05-09T05:01:20.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.099+0000 7f513b13ef40 1 -- 172.21.15.60:0/816827644 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x5556f7631480 msgr2=0x5556f762d0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.099+0000 7f513b13ef40 1 --2- 172.21.15.60:0/816827644 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x5556f7631480 0x5556f762d0e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.099+0000 7f513b13ef40 1 -- 172.21.15.60:0/816827644 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f511c074ce0 msgr2=0x7f511c0771a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.099+0000 7f513b13ef40 1 --2- 172.21.15.60:0/816827644 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f511c074ce0 0x7f511c0771a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5124056000 tx=0x7f5124051000 comp rx=0 tx=0).stop 2024-05-09T05:01:20.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.099+0000 7f513b13ef40 1 -- 172.21.15.60:0/816827644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5556f756be80 msgr2=0x5556f7630f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.099+0000 7f513b13ef40 1 --2- 172.21.15.60:0/816827644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5556f756be80 0x5556f7630f40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f512c001eb0 tx=0x7f512c001a60 comp rx=0 tx=0).stop 2024-05-09T05:01:20.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.099+0000 7f5139439640 1 -- 172.21.15.60:0/816827644 reap_dead start 2024-05-09T05:01:20.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.099+0000 7f513b13ef40 1 -- 172.21.15.60:0/816827644 shutdown_connections 2024-05-09T05:01:20.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.099+0000 7f513b13ef40 1 -- 172.21.15.60:0/816827644 >> 172.21.15.60:0/816827644 conn(0x5556f75948d0 msgr2=0x5556f7595ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:20.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.099+0000 7f513b13ef40 1 -- 172.21.15.60:0/816827644 shutdown_connections 2024-05-09T05:01:20.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.099+0000 7f513b13ef40 1 -- 172.21.15.60:0/816827644 wait complete. 2024-05-09T05:01:20.105 INFO:teuthology.orchestra.run.smithi060.stdout:a��alphaparent_b 2024-05-09T05:01:20.105 DEBUG:teuthology.orchestra.run.smithi060:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-05-09T05:01:20.154 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:01:20.154 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628283, 2024-05-09T05:01:20.154 INFO:teuthology.orchestra.run.smithi060.stdout: "ancestors": [ 2024-05-09T05:01:20.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:20.154 INFO:teuthology.orchestra.run.smithi060.stdout: "dirino": 1099511628284, 2024-05-09T05:01:20.154 INFO:teuthology.orchestra.run.smithi060.stdout: "dname": "alpha", 2024-05-09T05:01:20.154 INFO:teuthology.orchestra.run.smithi060.stdout: "version": 7 2024-05-09T05:01:20.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:20.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:20.155 INFO:teuthology.orchestra.run.smithi060.stdout: "dirino": 1, 2024-05-09T05:01:20.155 INFO:teuthology.orchestra.run.smithi060.stdout: "dname": "parent_b", 2024-05-09T05:01:20.155 INFO:teuthology.orchestra.run.smithi060.stdout: "version": 30 2024-05-09T05:01:20.155 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:20.155 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-05-09T05:01:20.155 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 7, 2024-05-09T05:01:20.155 INFO:teuthology.orchestra.run.smithi060.stdout: "old_pools": [ 2024-05-09T05:01:20.155 INFO:teuthology.orchestra.run.smithi060.stdout: 6 2024-05-09T05:01:20.155 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-05-09T05:01:20.155 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:01:20.155 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:20.156 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool data_new getxattr 100000001fb.00000000 parent 2024-05-09T05:01:20.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.240+0000 7f658fb4df40 1 Processor -- start 2024-05-09T05:01:20.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.240+0000 7f658fb4df40 1 -- start start 2024-05-09T05:01:20.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.241+0000 7f658fb4df40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55560dbd6690 0x55560dc7ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.241+0000 7f658fb4df40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55560dbeee30 0x55560dc72c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.241+0000 7f658fb4df40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55560dc7b760 0x55560dc7d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.241+0000 7f658fb4df40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55560dc7bb40 con 0x55560dc7b760 2024-05-09T05:01:20.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.241+0000 7f658fb4df40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55560db4df50 con 0x55560dbeee30 2024-05-09T05:01:20.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.241+0000 7f658fb4df40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x55560dbef330 con 0x55560dbd6690 2024-05-09T05:01:20.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.241+0000 7f658de48640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55560dc7b760 0x55560dc7d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.241+0000 7f658de48640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55560dc7b760 0x55560dc7d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59440/0 (socket says 172.21.15.60:59440) 2024-05-09T05:01:20.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.241+0000 7f658de48640 1 -- 172.21.15.60:0/1117839700 learned_addr learned my addr 172.21.15.60:0/1117839700 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:20.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.241+0000 7f658d647640 1 --2- 172.21.15.60:0/1117839700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55560dbd6690 0x55560dc7ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.241+0000 7f658ce46640 1 --2- 172.21.15.60:0/1117839700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55560dbeee30 0x55560dc72c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f658de48640 1 -- 172.21.15.60:0/1117839700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55560dbd6690 msgr2=0x55560dc7ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:20.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f658de48640 1 --2- 172.21.15.60:0/1117839700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55560dbd6690 0x55560dc7ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f658de48640 1 -- 172.21.15.60:0/1117839700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55560dbeee30 msgr2=0x55560dc72c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:20.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f658de48640 1 --2- 172.21.15.60:0/1117839700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55560dbeee30 0x55560dc72c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f658de48640 1 -- 172.21.15.60:0/1117839700 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55560dc7d3b0 con 0x55560dc7b760 2024-05-09T05:01:20.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f658ce46640 1 --2- 172.21.15.60:0/1117839700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55560dbeee30 0x55560dc72c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:20.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f658d647640 1 --2- 172.21.15.60:0/1117839700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55560dbd6690 0x55560dc7ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:20.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f658de48640 1 --2- 172.21.15.60:0/1117839700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55560dc7b760 0x55560dc7d9f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6588005fd0 tx=0x7f6588050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f5312033055169 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:20.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f657f7fe640 1 -- 172.21.15.60:0/1117839700 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6588054040 con 0x55560dc7b760 2024-05-09T05:01:20.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f657f7fe640 1 -- 172.21.15.60:0/1117839700 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6588050960 con 0x55560dc7b760 2024-05-09T05:01:20.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f657f7fe640 1 -- 172.21.15.60:0/1117839700 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6588050d30 con 0x55560dc7b760 2024-05-09T05:01:20.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f658fb4df40 1 -- 172.21.15.60:0/1117839700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55560dc7b760 msgr2=0x55560dc7d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f658fb4df40 1 --2- 172.21.15.60:0/1117839700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55560dc7b760 0x55560dc7d9f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6588005fd0 tx=0x7f6588050000 comp rx=0 tx=0).stop 2024-05-09T05:01:20.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f658fb4df40 1 -- 172.21.15.60:0/1117839700 shutdown_connections 2024-05-09T05:01:20.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f658fb4df40 1 --2- 172.21.15.60:0/1117839700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55560dc7b760 0x55560dc7d9f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f658fb4df40 1 --2- 172.21.15.60:0/1117839700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55560dbeee30 0x55560dc72c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f658fb4df40 1 --2- 172.21.15.60:0/1117839700 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55560dbd6690 0x55560dc7ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f658fb4df40 1 -- 172.21.15.60:0/1117839700 >> 172.21.15.60:0/1117839700 conn(0x55560dba6110 msgr2=0x55560dba64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:20.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f658fb4df40 1 -- 172.21.15.60:0/1117839700 shutdown_connections 2024-05-09T05:01:20.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.242+0000 7f658fb4df40 1 -- 172.21.15.60:0/1117839700 wait complete. 2024-05-09T05:01:20.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.243+0000 7f658fb4df40 1 Processor -- start 2024-05-09T05:01:20.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.243+0000 7f658fb4df40 1 -- start start 2024-05-09T05:01:20.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f658fb4df40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55560dbd6690 0x55560dd12880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f658fb4df40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55560dbeee30 0x55560dc6f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f658fb4df40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55560dc7b760 0x55560dc756c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f658fb4df40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55560dc78ea0 con 0x55560dbeee30 2024-05-09T05:01:20.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f658fb4df40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55560dc7d160 con 0x55560dbd6690 2024-05-09T05:01:20.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f658fb4df40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x55560dc73240 con 0x55560dc7b760 2024-05-09T05:01:20.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f658ce46640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55560dbeee30 0x55560dc6f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f658ce46640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55560dbeee30 0x55560dc6f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59446/0 (socket says 172.21.15.60:59446) 2024-05-09T05:01:20.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f658ce46640 1 -- 172.21.15.60:0/2184402500 learned_addr learned my addr 172.21.15.60:0/2184402500 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:20.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f658d647640 1 --2- 172.21.15.60:0/2184402500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55560dbd6690 0x55560dd12880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f658de48640 1 --2- 172.21.15.60:0/2184402500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55560dc7b760 0x55560dc756c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f658ce46640 1 -- 172.21.15.60:0/2184402500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55560dc7b760 msgr2=0x55560dc756c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f658ce46640 1 --2- 172.21.15.60:0/2184402500 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55560dc7b760 0x55560dc756c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f658ce46640 1 -- 172.21.15.60:0/2184402500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55560dbd6690 msgr2=0x55560dd12880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f658ce46640 1 --2- 172.21.15.60:0/2184402500 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55560dbd6690 0x55560dd12880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f658ce46640 1 -- 172.21.15.60:0/2184402500 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55560dc79ed0 con 0x55560dbeee30 2024-05-09T05:01:20.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f658ce46640 1 --2- 172.21.15.60:0/2184402500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55560dbeee30 0x55560dc6f6a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f658400dfc0 tx=0x7f658400b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:20.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f657e7fc640 1 -- 172.21.15.60:0/2184402500 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6584002e60 con 0x55560dbeee30 2024-05-09T05:01:20.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f657e7fc640 1 -- 172.21.15.60:0/2184402500 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6584011640 con 0x55560dbeee30 2024-05-09T05:01:20.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.244+0000 7f658fb4df40 1 -- 172.21.15.60:0/2184402500 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55560dd12dc0 con 0x55560dbeee30 2024-05-09T05:01:20.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.245+0000 7f657e7fc640 1 -- 172.21.15.60:0/2184402500 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6584011a10 con 0x55560dbeee30 2024-05-09T05:01:20.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.245+0000 7f658fb4df40 1 -- 172.21.15.60:0/2184402500 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55560dc79bf0 con 0x55560dbeee30 2024-05-09T05:01:20.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.246+0000 7f657e7fc640 1 -- 172.21.15.60:0/2184402500 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6584011bb0 con 0x55560dbeee30 2024-05-09T05:01:20.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.246+0000 7f657e7fc640 1 --2- 172.21.15.60:0/2184402500 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6570074cb0 0x7f6570077170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.247+0000 7f657e7fc640 1 -- 172.21.15.60:0/2184402500 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6496+0+0 (secure 0 0 0) 0x7f65840966e0 con 0x55560dbeee30 2024-05-09T05:01:20.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.247+0000 7f658d647640 1 --2- 172.21.15.60:0/2184402500 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6570074cb0 0x7f6570077170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.247+0000 7f658fb4df40 1 --2- 172.21.15.60:0/2184402500 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x55560dc73aa0 0x55560dc77810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.247+0000 7f658fb4df40 1 -- 172.21.15.60:0/2184402500 --> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] -- osd_op(unknown.0.0:1 7.13 7:ca8fdbba:::100000001fb.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x55560dc77d50 con 0x55560dc73aa0 2024-05-09T05:01:20.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.247+0000 7f658de48640 1 --2- 172.21.15.60:0/2184402500 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x55560dc73aa0 0x55560dc77810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.247+0000 7f658d647640 1 --2- 172.21.15.60:0/2184402500 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6570074cb0 0x7f6570077170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f65800075e0 tx=0x7f658000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:20.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.247+0000 7f658de48640 1 --2- 172.21.15.60:0/2184402500 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x55560dc73aa0 0x55560dc77810 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:20.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.247+0000 7f658de48640 1 -- 172.21.15.60:0/2184402500 <== osd.0 v2:172.21.15.60:6800/1298086532 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (103) out=103b] v0'0 uv1 ondisk = 0) ==== 164+0+103 (crc 0 0 0) 0x7f6588056030 con 0x55560dc73aa0 2024-05-09T05:01:20.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.247+0000 7f658fb4df40 1 -- 172.21.15.60:0/2184402500 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x55560dc73aa0 msgr2=0x55560dc77810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.247+0000 7f658fb4df40 1 --2- 172.21.15.60:0/2184402500 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x55560dc73aa0 0x55560dc77810 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.248+0000 7f658fb4df40 1 -- 172.21.15.60:0/2184402500 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6570074cb0 msgr2=0x7f6570077170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.248+0000 7f658fb4df40 1 --2- 172.21.15.60:0/2184402500 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6570074cb0 0x7f6570077170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f65800075e0 tx=0x7f658000b040 comp rx=0 tx=0).stop 2024-05-09T05:01:20.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.248+0000 7f658fb4df40 1 -- 172.21.15.60:0/2184402500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55560dbeee30 msgr2=0x55560dc6f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.248+0000 7f658fb4df40 1 --2- 172.21.15.60:0/2184402500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55560dbeee30 0x55560dc6f6a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f658400dfc0 tx=0x7f658400b5b0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.248+0000 7f658de48640 1 -- 172.21.15.60:0/2184402500 reap_dead start 2024-05-09T05:01:20.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.248+0000 7f658fb4df40 1 -- 172.21.15.60:0/2184402500 shutdown_connections 2024-05-09T05:01:20.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.248+0000 7f658fb4df40 1 -- 172.21.15.60:0/2184402500 >> 172.21.15.60:0/2184402500 conn(0x55560dba6110 msgr2=0x55560dba64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:20.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.248+0000 7f658fb4df40 1 -- 172.21.15.60:0/2184402500 shutdown_connections 2024-05-09T05:01:20.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.248+0000 7f658fb4df40 1 -- 172.21.15.60:0/2184402500 wait complete. 2024-05-09T05:01:20.253 INFO:teuthology.orchestra.run.smithi060.stdout:a��alphaparent_b 2024-05-09T05:01:20.254 DEBUG:teuthology.orchestra.run.smithi060:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-05-09T05:01:20.302 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:01:20.303 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628283, 2024-05-09T05:01:20.303 INFO:teuthology.orchestra.run.smithi060.stdout: "ancestors": [ 2024-05-09T05:01:20.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:20.303 INFO:teuthology.orchestra.run.smithi060.stdout: "dirino": 1099511628284, 2024-05-09T05:01:20.303 INFO:teuthology.orchestra.run.smithi060.stdout: "dname": "alpha", 2024-05-09T05:01:20.303 INFO:teuthology.orchestra.run.smithi060.stdout: "version": 7 2024-05-09T05:01:20.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:20.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:20.303 INFO:teuthology.orchestra.run.smithi060.stdout: "dirino": 1, 2024-05-09T05:01:20.303 INFO:teuthology.orchestra.run.smithi060.stdout: "dname": "parent_b", 2024-05-09T05:01:20.303 INFO:teuthology.orchestra.run.smithi060.stdout: "version": 30 2024-05-09T05:01:20.304 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:20.304 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-05-09T05:01:20.304 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 7, 2024-05-09T05:01:20.304 INFO:teuthology.orchestra.run.smithi060.stdout: "old_pools": [ 2024-05-09T05:01:20.304 INFO:teuthology.orchestra.run.smithi060.stdout: 6 2024-05-09T05:01:20.304 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-05-09T05:01:20.304 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:01:20.304 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:20.305 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool data_new getxattr 100000001fb.00000000 layout 2024-05-09T05:01:20.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.389+0000 7fe0d8222f40 1 Processor -- start 2024-05-09T05:01:20.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.390+0000 7fe0d8222f40 1 -- start start 2024-05-09T05:01:20.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.390+0000 7fe0d8222f40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8b8363d30 0x55f8b834ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.390+0000 7fe0d8222f40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55f8b83ef250 0x55f8b8364950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.390+0000 7fe0d8222f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f8b83f2b00 0x55f8b8364e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.390+0000 7fe0d8222f40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f8b8364330 con 0x55f8b8363d30 2024-05-09T05:01:20.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.390+0000 7fe0d8222f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f8b82c3f50 con 0x55f8b83f2b00 2024-05-09T05:01:20.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.390+0000 7fe0d8222f40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x55f8b83f2020 con 0x55f8b83ef250 2024-05-09T05:01:20.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.390+0000 7fe0d5d1e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8b8363d30 0x55f8b834ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.390+0000 7fe0d5d1e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8b8363d30 0x55f8b834ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59458/0 (socket says 172.21.15.60:59458) 2024-05-09T05:01:20.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.390+0000 7fe0d5d1e640 1 -- 172.21.15.60:0/804251740 learned_addr learned my addr 172.21.15.60:0/804251740 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:20.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0d551d640 1 --2- 172.21.15.60:0/804251740 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55f8b83ef250 0x55f8b8364950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0d651f640 1 --2- 172.21.15.60:0/804251740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f8b83f2b00 0x55f8b8364e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0d5d1e640 1 -- 172.21.15.60:0/804251740 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55f8b83ef250 msgr2=0x55f8b8364950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:20.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0d5d1e640 1 --2- 172.21.15.60:0/804251740 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55f8b83ef250 0x55f8b8364950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0d5d1e640 1 -- 172.21.15.60:0/804251740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f8b83f2b00 msgr2=0x55f8b8364e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:20.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0d5d1e640 1 --2- 172.21.15.60:0/804251740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f8b83f2b00 0x55f8b8364e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0d5d1e640 1 -- 172.21.15.60:0/804251740 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8b834cea0 con 0x55f8b8363d30 2024-05-09T05:01:20.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0d5d1e640 1 --2- 172.21.15.60:0/804251740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8b8363d30 0x55f8b834ba20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe0c80096b0 tx=0x7fe0c8009b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67d5ce4a6c975b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:20.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0c7fff640 1 -- 172.21.15.60:0/804251740 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c8014070 con 0x55f8b8363d30 2024-05-09T05:01:20.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0c7fff640 1 -- 172.21.15.60:0/804251740 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0c800b930 con 0x55f8b8363d30 2024-05-09T05:01:20.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0c7fff640 1 -- 172.21.15.60:0/804251740 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c800bd20 con 0x55f8b8363d30 2024-05-09T05:01:20.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0d8222f40 1 -- 172.21.15.60:0/804251740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8b8363d30 msgr2=0x55f8b834ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0d8222f40 1 --2- 172.21.15.60:0/804251740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8b8363d30 0x55f8b834ba20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe0c80096b0 tx=0x7fe0c8009b70 comp rx=0 tx=0).stop 2024-05-09T05:01:20.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0d8222f40 1 -- 172.21.15.60:0/804251740 shutdown_connections 2024-05-09T05:01:20.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0d8222f40 1 --2- 172.21.15.60:0/804251740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f8b83f2b00 0x55f8b8364e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0d8222f40 1 --2- 172.21.15.60:0/804251740 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55f8b83ef250 0x55f8b8364950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0d8222f40 1 --2- 172.21.15.60:0/804251740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8b8363d30 0x55f8b834ba20 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0d8222f40 1 -- 172.21.15.60:0/804251740 >> 172.21.15.60:0/804251740 conn(0x55f8b83468d0 msgr2=0x55f8b831d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:20.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0d8222f40 1 -- 172.21.15.60:0/804251740 shutdown_connections 2024-05-09T05:01:20.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.391+0000 7fe0d8222f40 1 -- 172.21.15.60:0/804251740 wait complete. 2024-05-09T05:01:20.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.392+0000 7fe0d8222f40 1 Processor -- start 2024-05-09T05:01:20.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.392+0000 7fe0d8222f40 1 -- start start 2024-05-09T05:01:20.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.393+0000 7fe0d8222f40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8b8363d30 0x55f8b83e3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.393+0000 7fe0d8222f40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55f8b83ef250 0x55f8b83de720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.393+0000 7fe0d8222f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f8b83f2b00 0x55f8b8346cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.393+0000 7fe0d8222f40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55f8b83e67c0 con 0x55f8b8363d30 2024-05-09T05:01:20.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.393+0000 7fe0d8222f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f8b83f16e0 con 0x55f8b83f2b00 2024-05-09T05:01:20.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.393+0000 7fe0d8222f40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x55f8b83e9f80 con 0x55f8b83ef250 2024-05-09T05:01:20.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.393+0000 7fe0d5d1e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8b8363d30 0x55f8b83e3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.393+0000 7fe0d5d1e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8b8363d30 0x55f8b83e3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59470/0 (socket says 172.21.15.60:59470) 2024-05-09T05:01:20.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.393+0000 7fe0d5d1e640 1 -- 172.21.15.60:0/2852219485 learned_addr learned my addr 172.21.15.60:0/2852219485 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.393+0000 7fe0d651f640 1 --2- 172.21.15.60:0/2852219485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f8b83f2b00 0x55f8b8346cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.393+0000 7fe0d551d640 1 --2- 172.21.15.60:0/2852219485 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55f8b83ef250 0x55f8b83de720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.393+0000 7fe0d5d1e640 1 -- 172.21.15.60:0/2852219485 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55f8b83ef250 msgr2=0x55f8b83de720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.393+0000 7fe0d5d1e640 1 --2- 172.21.15.60:0/2852219485 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55f8b83ef250 0x55f8b83de720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.393+0000 7fe0d5d1e640 1 -- 172.21.15.60:0/2852219485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f8b83f2b00 msgr2=0x55f8b8346cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.393+0000 7fe0d5d1e640 1 --2- 172.21.15.60:0/2852219485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f8b83f2b00 0x55f8b8346cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.393+0000 7fe0d5d1e640 1 -- 172.21.15.60:0/2852219485 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8b83f49f0 con 0x55f8b8363d30 2024-05-09T05:01:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.393+0000 7fe0d5d1e640 1 --2- 172.21.15.60:0/2852219485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8b8363d30 0x55f8b83e3160 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8009bb0 tx=0x7fe0c80033a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.393+0000 7fe0c6ffd640 1 -- 172.21.15.60:0/2852219485 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c8029070 con 0x55f8b8363d30 2024-05-09T05:01:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.393+0000 7fe0c6ffd640 1 -- 172.21.15.60:0/2852219485 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0c8019740 con 0x55f8b8363d30 2024-05-09T05:01:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.394+0000 7fe0d8222f40 1 -- 172.21.15.60:0/2852219485 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f8b8346120 con 0x55f8b8363d30 2024-05-09T05:01:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.394+0000 7fe0c6ffd640 1 -- 172.21.15.60:0/2852219485 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c8014040 con 0x55f8b8363d30 2024-05-09T05:01:20.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.394+0000 7fe0d8222f40 1 -- 172.21.15.60:0/2852219485 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8b83e6d40 con 0x55f8b8363d30 2024-05-09T05:01:20.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.395+0000 7fe0c6ffd640 1 -- 172.21.15.60:0/2852219485 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe0c802c070 con 0x55f8b8363d30 2024-05-09T05:01:20.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.395+0000 7fe0c6ffd640 1 --2- 172.21.15.60:0/2852219485 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe0b8074cb0 0x7fe0b8077170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.396+0000 7fe0c6ffd640 1 -- 172.21.15.60:0/2852219485 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6496+0+0 (secure 0 0 0) 0x7fe0c8097b90 con 0x55f8b8363d30 2024-05-09T05:01:20.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.396+0000 7fe0d551d640 1 --2- 172.21.15.60:0/2852219485 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe0b8074cb0 0x7fe0b8077170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.396+0000 7fe0d8222f40 1 --2- 172.21.15.60:0/2852219485 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x55f8b83476a0 0x55f8b83e87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.396+0000 7fe0d8222f40 1 -- 172.21.15.60:0/2852219485 --> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] -- osd_op(unknown.0.0:1 7.13 7:ca8fdbba:::100000001fb.00000000:head [getxattr layout in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x55f8b83e8d10 con 0x55f8b83476a0 2024-05-09T05:01:20.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.396+0000 7fe0d651f640 1 --2- 172.21.15.60:0/2852219485 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x55f8b83476a0 0x55f8b83e87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.396+0000 7fe0d551d640 1 --2- 172.21.15.60:0/2852219485 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe0b8074cb0 0x7fe0b8077170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55f8b83f4df0 tx=0x7fe0cc00e040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:20.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.396+0000 7fe0d651f640 1 --2- 172.21.15.60:0/2852219485 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x55f8b83476a0 0x55f8b83e87d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:20.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.396+0000 7fe0d651f640 1 -- 172.21.15.60:0/2852219485 <== osd.0 v2:172.21.15.60:6800/1298086532 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (30) out=30b] v0'0 uv1 ondisk = 0) ==== 164+0+30 (crc 0 0 0) 0x7fe0d0051070 con 0x55f8b83476a0 2024-05-09T05:01:20.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.397+0000 7fe0d8222f40 1 -- 172.21.15.60:0/2852219485 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x55f8b83476a0 msgr2=0x55f8b83e87d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.397+0000 7fe0d8222f40 1 --2- 172.21.15.60:0/2852219485 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x55f8b83476a0 0x55f8b83e87d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.397+0000 7fe0d8222f40 1 -- 172.21.15.60:0/2852219485 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe0b8074cb0 msgr2=0x7fe0b8077170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.397+0000 7fe0d8222f40 1 --2- 172.21.15.60:0/2852219485 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe0b8074cb0 0x7fe0b8077170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55f8b83f4df0 tx=0x7fe0cc00e040 comp rx=0 tx=0).stop 2024-05-09T05:01:20.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.397+0000 7fe0d8222f40 1 -- 172.21.15.60:0/2852219485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8b8363d30 msgr2=0x55f8b83e3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.397+0000 7fe0d8222f40 1 --2- 172.21.15.60:0/2852219485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8b8363d30 0x55f8b83e3160 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8009bb0 tx=0x7fe0c80033a0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.397+0000 7fe0d651f640 1 -- 172.21.15.60:0/2852219485 reap_dead start 2024-05-09T05:01:20.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.397+0000 7fe0d8222f40 1 -- 172.21.15.60:0/2852219485 shutdown_connections 2024-05-09T05:01:20.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.397+0000 7fe0d8222f40 1 -- 172.21.15.60:0/2852219485 >> 172.21.15.60:0/2852219485 conn(0x55f8b83468d0 msgr2=0x55f8b8363410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:20.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.397+0000 7fe0d8222f40 1 -- 172.21.15.60:0/2852219485 shutdown_connections 2024-05-09T05:01:20.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.397+0000 7fe0d8222f40 1 -- 172.21.15.60:0/2852219485 wait complete. 2024-05-09T05:01:20.403 INFO:teuthology.orchestra.run.smithi060.stdout:@@ 2024-05-09T05:01:20.403 DEBUG:teuthology.orchestra.run.smithi060:> ceph-dencoder type file_layout_t import - decode dump_json 2024-05-09T05:01:20.452 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:01:20.452 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_unit": 4194304, 2024-05-09T05:01:20.452 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_count": 1, 2024-05-09T05:01:20.452 INFO:teuthology.orchestra.run.smithi060.stdout: "object_size": 4194304, 2024-05-09T05:01:20.452 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_id": 7, 2024-05-09T05:01:20.452 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_ns": "" 2024-05-09T05:01:20.452 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:01:20.452 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:20.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:20.454 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent_c) 2024-05-09T05:01:20.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:20.512 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mv parent_b/alpha parent_c/alpha) 2024-05-09T05:01:20.572 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:20.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.716+0000 7f0720a85640 1 Processor -- start 2024-05-09T05:01:20.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.716+0000 7f0720a85640 1 -- start start 2024-05-09T05:01:20.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.717+0000 7f0720a85640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f071c15f300 0x7f071c072220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.717+0000 7f0720a85640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f071c06fe20 0x7f071c0702a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.717+0000 7f0720a85640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f071c072760 con 0x7f071c0736d0 2024-05-09T05:01:20.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.717+0000 7f0720a85640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f071c073240 con 0x7f071c15f300 2024-05-09T05:01:20.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.717+0000 7f0720a85640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f071c0d08e0 con 0x7f071c06fe20 2024-05-09T05:01:20.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.717+0000 7f071a59c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f071c15f300 0x7f071c072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.717+0000 7f071ad9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f071c0736d0 0x7f071c073b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42058/0 (socket says 172.21.15.60:42058) 2024-05-09T05:01:20.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.717+0000 7f071ad9d640 1 -- 172.21.15.60:0/2378405770 learned_addr learned my addr 172.21.15.60:0/2378405770 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:20.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.718+0000 7f0719d9b640 1 --2- 172.21.15.60:0/2378405770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f071c06fe20 0x7f071c0702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.718+0000 7f071a59c640 1 -- 172.21.15.60:0/2378405770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f071c06fe20 msgr2=0x7f071c0702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:20.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.718+0000 7f071a59c640 1 --2- 172.21.15.60:0/2378405770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f071c06fe20 0x7f071c0702a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.718+0000 7f071a59c640 1 -- 172.21.15.60:0/2378405770 >> v1:172.21.15.60:6789/0 conn(0x7f071c0736d0 legacy=0x7f071c073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.718+0000 7f071a59c640 1 -- 172.21.15.60:0/2378405770 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071c0d0bc0 con 0x7f071c15f300 2024-05-09T05:01:20.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.718+0000 7f071959a640 1 -- 172.21.15.60:0/2378405770 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2583994663 0 0) 0x7f0714001110 con 0x7f071c0736d0 2024-05-09T05:01:20.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.718+0000 7f071a59c640 1 --2- 172.21.15.60:0/2378405770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c15f300 0x7f071c072220 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f070c009d90 tx=0x7f070c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ff8c2b8e98e4d1b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:20.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.719+0000 7f071959a640 1 -- 172.21.15.60:0/2378405770 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070c007740 con 0x7f071c15f300 2024-05-09T05:01:20.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.719+0000 7f071959a640 1 -- 172.21.15.60:0/2378405770 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f070c0078e0 con 0x7f071c15f300 2024-05-09T05:01:20.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.719+0000 7f071959a640 1 -- 172.21.15.60:0/2378405770 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070c007cb0 con 0x7f071c15f300 2024-05-09T05:01:20.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.719+0000 7f0720a85640 1 -- 172.21.15.60:0/2378405770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c15f300 msgr2=0x7f071c072220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.719+0000 7f0720a85640 1 --2- 172.21.15.60:0/2378405770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c15f300 0x7f071c072220 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f070c009d90 tx=0x7f070c02a660 comp rx=0 tx=0).stop 2024-05-09T05:01:20.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.719+0000 7f0720a85640 1 -- 172.21.15.60:0/2378405770 shutdown_connections 2024-05-09T05:01:20.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.719+0000 7f0720a85640 1 --2- 172.21.15.60:0/2378405770 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f071c06fe20 0x7f071c0702a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.719+0000 7f0720a85640 1 --2- 172.21.15.60:0/2378405770 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c15f300 0x7f071c072220 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.719+0000 7f0720a85640 1 -- 172.21.15.60:0/2378405770 >> 172.21.15.60:0/2378405770 conn(0x7f071c0b4ef0 msgr2=0x7f071c06e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:20.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.719+0000 7f0720a85640 1 -- 172.21.15.60:0/2378405770 shutdown_connections 2024-05-09T05:01:20.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.719+0000 7f0720a85640 1 -- 172.21.15.60:0/2378405770 wait complete. 2024-05-09T05:01:20.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.721+0000 7f0720a85640 1 Processor -- start 2024-05-09T05:01:20.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.721+0000 7f0720a85640 1 -- start start 2024-05-09T05:01:20.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.721+0000 7f0720a85640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c06fe20 0x7f071c0af630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.721+0000 7f0720a85640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f071c0736d0 0x7f071c1f4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.721+0000 7f0720a85640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f071c15f300 0x7f071c1f51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.721+0000 7f0720a85640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f071c0d04f0 con 0x7f071c06fe20 2024-05-09T05:01:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.721+0000 7f0720a85640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f071c09c1e0 con 0x7f071c15f300 2024-05-09T05:01:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.721+0000 7f0720a85640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f071c0d0670 con 0x7f071c0736d0 2024-05-09T05:01:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.721+0000 7f071a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c06fe20 0x7f071c0af630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.721+0000 7f071a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c06fe20 0x7f071c0af630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59496/0 (socket says 172.21.15.60:59496) 2024-05-09T05:01:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.722+0000 7f071a59c640 1 -- 172.21.15.60:0/1326595484 learned_addr learned my addr 172.21.15.60:0/1326595484 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.722+0000 7f0719d9b640 1 --2- 172.21.15.60:0/1326595484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f071c0736d0 0x7f071c1f4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.722+0000 7f071ad9d640 1 --2- 172.21.15.60:0/1326595484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f071c15f300 0x7f071c1f51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.722+0000 7f071a59c640 1 -- 172.21.15.60:0/1326595484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f071c0736d0 msgr2=0x7f071c1f4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.722+0000 7f071a59c640 1 --2- 172.21.15.60:0/1326595484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f071c0736d0 0x7f071c1f4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.722+0000 7f071a59c640 1 -- 172.21.15.60:0/1326595484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f071c15f300 msgr2=0x7f071c1f51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.722+0000 7f071a59c640 1 --2- 172.21.15.60:0/1326595484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f071c15f300 0x7f071c1f51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.722+0000 7f071a59c640 1 -- 172.21.15.60:0/1326595484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071c0ae1d0 con 0x7f071c06fe20 2024-05-09T05:01:20.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.722+0000 7f071a59c640 1 --2- 172.21.15.60:0/1326595484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c06fe20 0x7f071c0af630 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f070c009d90 tx=0x7f070c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:20.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.722+0000 7f0712ffd640 1 -- 172.21.15.60:0/1326595484 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070c03f070 con 0x7f071c06fe20 2024-05-09T05:01:20.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.722+0000 7f0712ffd640 1 -- 172.21.15.60:0/1326595484 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f070c031c40 con 0x7f071c06fe20 2024-05-09T05:01:20.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.722+0000 7f0720a85640 1 -- 172.21.15.60:0/1326595484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f071c0ae4b0 con 0x7f071c06fe20 2024-05-09T05:01:20.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.722+0000 7f0712ffd640 1 -- 172.21.15.60:0/1326595484 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070c043410 con 0x7f071c06fe20 2024-05-09T05:01:20.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.722+0000 7f0720a85640 1 -- 172.21.15.60:0/1326595484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f071c0aea70 con 0x7f071c06fe20 2024-05-09T05:01:20.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.723+0000 7f0720a85640 1 -- 172.21.15.60:0/1326595484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f071c09b820 con 0x7f071c06fe20 2024-05-09T05:01:20.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.724+0000 7f0712ffd640 1 -- 172.21.15.60:0/1326595484 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f070c0435b0 con 0x7f071c06fe20 2024-05-09T05:01:20.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.724+0000 7f0712ffd640 1 --2- 172.21.15.60:0/1326595484 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f06fc0762d0 0x7f06fc078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:20.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.725+0000 7f0712ffd640 1 -- 172.21.15.60:0/1326595484 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6496+0+0 (secure 0 0 0) 0x7f070c0b8d00 con 0x7f071c06fe20 2024-05-09T05:01:20.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.725+0000 7f0719d9b640 1 --2- 172.21.15.60:0/1326595484 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f06fc0762d0 0x7f06fc078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:20.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.725+0000 7f0719d9b640 1 --2- 172.21.15.60:0/1326595484 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f06fc0762d0 0x7f06fc078790 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f071c15cd60 tx=0x7f0704008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:20.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.729+0000 7f0712ffd640 1 -- 172.21.15.60:0/1326595484 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f071c09b820 con 0x7f071c06fe20 2024-05-09T05:01:20.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.961+0000 7f0720a85640 1 -- 172.21.15.60:0/1326595484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f071c075aa0 con 0x7f071c06fe20 2024-05-09T05:01:20.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.962+0000 7f0712ffd640 1 -- 172.21.15.60:0/1326595484 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5183 (secure 0 0 0) 0x7f071c075aa0 con 0x7f071c06fe20 2024-05-09T05:01:20.962 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:20.963 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4547,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2372252858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2372252858},{"type":"v1","addr":"172.21.15.114:6836","nonce":2372252858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1761282610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1761282610},{"type":"v1","addr":"172.21.15.114:6837","nonce":1761282610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4765,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/300408012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":300408012},{"type":"v1","addr":"172.21.15.60:6835","nonce":300408012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:04.698975+0000","modified":"2024-05-09T05:01:18.144520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4893},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4893":{"gid":4893,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/399160914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":399160914},{"type":"v1","addr":"172.21.15.60:6834","nonce":399160914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4893,"qdb_cluster":[4893]},"id":2}]} 2024-05-09T05:01:20.963 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 18 2024-05-09T05:01:20.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.964+0000 7f0720a85640 1 -- 172.21.15.60:0/1326595484 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f06fc0762d0 msgr2=0x7f06fc078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.964+0000 7f0720a85640 1 --2- 172.21.15.60:0/1326595484 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f06fc0762d0 0x7f06fc078790 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f071c15cd60 tx=0x7f0704008040 comp rx=0 tx=0).stop 2024-05-09T05:01:20.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.964+0000 7f0720a85640 1 -- 172.21.15.60:0/1326595484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c06fe20 msgr2=0x7f071c0af630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:20.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.964+0000 7f0720a85640 1 --2- 172.21.15.60:0/1326595484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c06fe20 0x7f071c0af630 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f070c009d90 tx=0x7f070c004a40 comp rx=0 tx=0).stop 2024-05-09T05:01:20.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.964+0000 7f0720a85640 1 -- 172.21.15.60:0/1326595484 shutdown_connections 2024-05-09T05:01:20.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.964+0000 7f0720a85640 1 --2- 172.21.15.60:0/1326595484 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f06fc0762d0 0x7f06fc078790 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.964+0000 7f0720a85640 1 --2- 172.21.15.60:0/1326595484 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f071c15f300 0x7f071c1f51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.964+0000 7f0720a85640 1 --2- 172.21.15.60:0/1326595484 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f071c0736d0 0x7f071c1f4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.964+0000 7f0720a85640 1 --2- 172.21.15.60:0/1326595484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c06fe20 0x7f071c0af630 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:20.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.964+0000 7f0720a85640 1 -- 172.21.15.60:0/1326595484 >> 172.21.15.60:0/1326595484 conn(0x7f071c0b4ef0 msgr2=0x7f071c161eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:20.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.964+0000 7f0720a85640 1 -- 172.21.15.60:0/1326595484 shutdown_connections 2024-05-09T05:01:20.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:20.964+0000 7f0720a85640 1 -- 172.21.15.60:0/1326595484 wait complete. 2024-05-09T05:01:20.975 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-05-09T05:01:21.104 INFO:teuthology.orchestra.run.smithi060.stdout:{"message":"","return_code":0} 2024-05-09T05:01:21.104 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-05-09T05:01:21.104 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001fb.00000000 parent 2024-05-09T05:01:21.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.149+0000 7fb70ef1bf40 1 Processor -- start 2024-05-09T05:01:21.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.149+0000 7fb70ef1bf40 1 -- start start 2024-05-09T05:01:21.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.149+0000 7fb70ef1bf40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bd97a62c0 0x560bd9834220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.149+0000 7fb70ef1bf40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560bd9833480 0x560bd975da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.149+0000 7fb70ef1bf40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bd9833860 0x560bd975dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.149+0000 7fb70ef1bf40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x560bd978f2f0 con 0x560bd9833860 2024-05-09T05:01:21.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.149+0000 7fb70ef1bf40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560bd9706f50 con 0x560bd97a62c0 2024-05-09T05:01:21.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.149+0000 7fb70ef1bf40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x560bd978f850 con 0x560bd9833480 2024-05-09T05:01:21.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.150+0000 7fb70c8ed640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bd9833860 0x560bd975dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.150+0000 7fb70c8ed640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bd9833860 0x560bd975dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59518/0 (socket says 172.21.15.60:59518) 2024-05-09T05:01:21.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.150+0000 7fb70c8ed640 1 -- 172.21.15.60:0/1694949821 learned_addr learned my addr 172.21.15.60:0/1694949821 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:21.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.150+0000 7fb70c0ec640 1 --2- 172.21.15.60:0/1694949821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bd97a62c0 0x560bd9834220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.150+0000 7fb70b8eb640 1 --2- 172.21.15.60:0/1694949821 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560bd9833480 0x560bd975da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.150+0000 7fb70c8ed640 1 -- 172.21.15.60:0/1694949821 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560bd9833480 msgr2=0x560bd975da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:21.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.150+0000 7fb70c8ed640 1 --2- 172.21.15.60:0/1694949821 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560bd9833480 0x560bd975da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.150+0000 7fb70c8ed640 1 -- 172.21.15.60:0/1694949821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bd97a62c0 msgr2=0x560bd9834220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:21.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.150+0000 7fb70c8ed640 1 --2- 172.21.15.60:0/1694949821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bd97a62c0 0x560bd9834220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.150+0000 7fb70c8ed640 1 -- 172.21.15.60:0/1694949821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bd97a68a0 con 0x560bd9833860 2024-05-09T05:01:21.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.150+0000 7fb70b8eb640 1 --2- 172.21.15.60:0/1694949821 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560bd9833480 0x560bd975da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:21.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.150+0000 7fb70c8ed640 1 --2- 172.21.15.60:0/1694949821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bd9833860 0x560bd975dfc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb70400b430 tx=0x7fb70400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=584b21e5f811dfa6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:21.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.150+0000 7fb70a8e9640 1 -- 172.21.15.60:0/1694949821 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb704008070 con 0x560bd9833860 2024-05-09T05:01:21.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.150+0000 7fb70a8e9640 1 -- 172.21.15.60:0/1694949821 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb70400d5c0 con 0x560bd9833860 2024-05-09T05:01:21.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.150+0000 7fb70a8e9640 1 -- 172.21.15.60:0/1694949821 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70400d9b0 con 0x560bd9833860 2024-05-09T05:01:21.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.151+0000 7fb70ef1bf40 1 -- 172.21.15.60:0/1694949821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bd9833860 msgr2=0x560bd975dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:21.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.151+0000 7fb70ef1bf40 1 --2- 172.21.15.60:0/1694949821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bd9833860 0x560bd975dfc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb70400b430 tx=0x7fb70400b8f0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.151+0000 7fb70ef1bf40 1 -- 172.21.15.60:0/1694949821 shutdown_connections 2024-05-09T05:01:21.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.151+0000 7fb70ef1bf40 1 --2- 172.21.15.60:0/1694949821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bd9833860 0x560bd975dfc0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.151+0000 7fb70ef1bf40 1 --2- 172.21.15.60:0/1694949821 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560bd9833480 0x560bd975da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.151+0000 7fb70ef1bf40 1 --2- 172.21.15.60:0/1694949821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bd97a62c0 0x560bd9834220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.151+0000 7fb70ef1bf40 1 -- 172.21.15.60:0/1694949821 >> 172.21.15.60:0/1694949821 conn(0x560bd97898d0 msgr2=0x560bd97a7050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:21.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.151+0000 7fb70ef1bf40 1 -- 172.21.15.60:0/1694949821 shutdown_connections 2024-05-09T05:01:21.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.151+0000 7fb70ef1bf40 1 -- 172.21.15.60:0/1694949821 wait complete. 2024-05-09T05:01:21.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.151+0000 7fb70ef1bf40 1 Processor -- start 2024-05-09T05:01:21.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.151+0000 7fb70ef1bf40 1 -- start start 2024-05-09T05:01:21.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.152+0000 7fb70ef1bf40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bd97a62c0 0x560bd9823d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.152+0000 7fb70ef1bf40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bd9833480 0x560bd9835020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.152+0000 7fb70ef1bf40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560bd9833860 0x560bd975da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.152+0000 7fb70ef1bf40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x560bd98c3180 con 0x560bd9833480 2024-05-09T05:01:21.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.152+0000 7fb70ef1bf40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560bd97a7970 con 0x560bd97a62c0 2024-05-09T05:01:21.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.152+0000 7fb70ef1bf40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x560bd9789440 con 0x560bd9833860 2024-05-09T05:01:21.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.152+0000 7fb70b8eb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bd9833480 0x560bd9835020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.152+0000 7fb70b8eb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bd9833480 0x560bd9835020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59528/0 (socket says 172.21.15.60:59528) 2024-05-09T05:01:21.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.152+0000 7fb70b8eb640 1 -- 172.21.15.60:0/3480826715 learned_addr learned my addr 172.21.15.60:0/3480826715 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:21.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.152+0000 7fb70c0ec640 1 --2- 172.21.15.60:0/3480826715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bd97a62c0 0x560bd9823d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.152+0000 7fb70c8ed640 1 --2- 172.21.15.60:0/3480826715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560bd9833860 0x560bd975da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.153+0000 7fb70b8eb640 1 -- 172.21.15.60:0/3480826715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560bd9833860 msgr2=0x560bd975da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:21.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.153+0000 7fb70b8eb640 1 --2- 172.21.15.60:0/3480826715 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x560bd9833860 0x560bd975da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.153+0000 7fb70b8eb640 1 -- 172.21.15.60:0/3480826715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bd97a62c0 msgr2=0x560bd9823d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:21.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.153+0000 7fb70b8eb640 1 --2- 172.21.15.60:0/3480826715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560bd97a62c0 0x560bd9823d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.153+0000 7fb70b8eb640 1 -- 172.21.15.60:0/3480826715 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bd9824260 con 0x560bd9833480 2024-05-09T05:01:21.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.153+0000 7fb70b8eb640 1 --2- 172.21.15.60:0/3480826715 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bd9833480 0x560bd9835020 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb700058040 tx=0x7fb700002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:21.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.153+0000 7fb7098e7640 1 -- 172.21.15.60:0/3480826715 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700064070 con 0x560bd9833480 2024-05-09T05:01:21.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.153+0000 7fb7098e7640 1 -- 172.21.15.60:0/3480826715 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb700055640 con 0x560bd9833480 2024-05-09T05:01:21.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.153+0000 7fb70ef1bf40 1 -- 172.21.15.60:0/3480826715 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560bd9789690 con 0x560bd9833480 2024-05-09T05:01:21.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.153+0000 7fb7098e7640 1 -- 172.21.15.60:0/3480826715 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70005a040 con 0x560bd9833480 2024-05-09T05:01:21.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.153+0000 7fb70ef1bf40 1 -- 172.21.15.60:0/3480826715 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bd9760db0 con 0x560bd9833480 2024-05-09T05:01:21.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.155+0000 7fb7098e7640 1 -- 172.21.15.60:0/3480826715 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb700067070 con 0x560bd9833480 2024-05-09T05:01:21.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.155+0000 7fb7098e7640 1 --2- 172.21.15.60:0/3480826715 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb6f8074cb0 0x7fb6f8077170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.155+0000 7fb7098e7640 1 -- 172.21.15.60:0/3480826715 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6496+0+0 (secure 0 0 0) 0x7fb7000dab30 con 0x560bd9833480 2024-05-09T05:01:21.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.155+0000 7fb70c0ec640 1 --2- 172.21.15.60:0/3480826715 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb6f8074cb0 0x7fb6f8077170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.155+0000 7fb70ef1bf40 1 --2- 172.21.15.60:0/3480826715 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x560bd9829480 0x560bd98cb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.155+0000 7fb70ef1bf40 1 -- 172.21.15.60:0/3480826715 --> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] -- osd_op(unknown.0.0:1 6.13 6:ca8fdbba:::100000001fb.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x560bd978b190 con 0x560bd9829480 2024-05-09T05:01:21.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.155+0000 7fb70c8ed640 1 --2- 172.21.15.60:0/3480826715 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x560bd9829480 0x560bd98cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.156+0000 7fb70c0ec640 1 --2- 172.21.15.60:0/3480826715 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb6f8074cb0 0x7fb6f8077170 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x560bd9832d30 tx=0x7fb6fc007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:21.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.156+0000 7fb70c8ed640 1 --2- 172.21.15.60:0/3480826715 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x560bd9829480 0x560bd98cb340 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:21.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.156+0000 7fb70c8ed640 1 -- 172.21.15.60:0/3480826715 <== osd.3 v2:172.21.15.60:6824/1069806621 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (103) out=103b] v0'0 uv3 ondisk = 0) ==== 164+0+103 (crc 0 0 0) 0x7fb70400d7f0 con 0x560bd9829480 2024-05-09T05:01:21.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.156+0000 7fb70ef1bf40 1 -- 172.21.15.60:0/3480826715 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x560bd9829480 msgr2=0x560bd98cb340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:21.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.156+0000 7fb70ef1bf40 1 --2- 172.21.15.60:0/3480826715 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x560bd9829480 0x560bd98cb340 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.156+0000 7fb70ef1bf40 1 -- 172.21.15.60:0/3480826715 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb6f8074cb0 msgr2=0x7fb6f8077170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:21.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.156+0000 7fb70ef1bf40 1 --2- 172.21.15.60:0/3480826715 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb6f8074cb0 0x7fb6f8077170 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x560bd9832d30 tx=0x7fb6fc007040 comp rx=0 tx=0).stop 2024-05-09T05:01:21.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.156+0000 7fb70ef1bf40 1 -- 172.21.15.60:0/3480826715 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bd9833480 msgr2=0x560bd9835020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:21.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.156+0000 7fb70ef1bf40 1 --2- 172.21.15.60:0/3480826715 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560bd9833480 0x560bd9835020 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb700058040 tx=0x7fb700002fb0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.156+0000 7fb70c8ed640 1 -- 172.21.15.60:0/3480826715 reap_dead start 2024-05-09T05:01:21.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.156+0000 7fb70ef1bf40 1 -- 172.21.15.60:0/3480826715 shutdown_connections 2024-05-09T05:01:21.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.156+0000 7fb70ef1bf40 1 -- 172.21.15.60:0/3480826715 >> 172.21.15.60:0/3480826715 conn(0x560bd97898d0 msgr2=0x560bd978aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:21.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.156+0000 7fb70ef1bf40 1 -- 172.21.15.60:0/3480826715 shutdown_connections 2024-05-09T05:01:21.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.156+0000 7fb70ef1bf40 1 -- 172.21.15.60:0/3480826715 wait complete. 2024-05-09T05:01:21.162 INFO:teuthology.orchestra.run.smithi060.stdout:a��alphaparent_b 2024-05-09T05:01:21.162 DEBUG:teuthology.orchestra.run.smithi060:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-05-09T05:01:21.211 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:01:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628283, 2024-05-09T05:01:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: "ancestors": [ 2024-05-09T05:01:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: "dirino": 1099511628284, 2024-05-09T05:01:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: "dname": "alpha", 2024-05-09T05:01:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: "version": 7 2024-05-09T05:01:21.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: "dirino": 1, 2024-05-09T05:01:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: "dname": "parent_b", 2024-05-09T05:01:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: "version": 30 2024-05-09T05:01:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-05-09T05:01:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 7, 2024-05-09T05:01:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: "old_pools": [ 2024-05-09T05:01:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: 6 2024-05-09T05:01:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-05-09T05:01:21.212 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:01:21.212 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:21.213 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool data_new getxattr 100000001fb.00000000 parent 2024-05-09T05:01:21.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.297+0000 7effa451af40 1 Processor -- start 2024-05-09T05:01:21.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.297+0000 7effa451af40 1 -- start start 2024-05-09T05:01:21.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.298+0000 7effa451af40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1c1e321f0 0x55c1c1e63680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.298+0000 7effa451af40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1c1e649d0 0x55c1c1eedf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.298+0000 7effa451af40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c1c1e7e770 0x55c1c1e61f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.298+0000 7effa451af40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c1c1ddbf50 con 0x55c1c1e7e770 2024-05-09T05:01:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.298+0000 7effa451af40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c1c1ddff90 con 0x55c1c1e649d0 2024-05-09T05:01:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.298+0000 7effa451af40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x55c1c1e64db0 con 0x55c1c1e321f0 2024-05-09T05:01:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.298+0000 7effa1db1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c1c1e7e770 0x55c1c1e61f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.298+0000 7effa1db1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c1c1e7e770 0x55c1c1e61f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59548/0 (socket says 172.21.15.60:59548) 2024-05-09T05:01:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.298+0000 7effa1db1640 1 -- 172.21.15.60:0/2817847155 learned_addr learned my addr 172.21.15.60:0/2817847155 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.298+0000 7effa0daf640 1 --2- 172.21.15.60:0/2817847155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1c1e649d0 0x55c1c1eedf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.298+0000 7effa15b0640 1 --2- 172.21.15.60:0/2817847155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1c1e321f0 0x55c1c1e63680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.299+0000 7effa1db1640 1 -- 172.21.15.60:0/2817847155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1c1e321f0 msgr2=0x55c1c1e63680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.299+0000 7effa1db1640 1 --2- 172.21.15.60:0/2817847155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1c1e321f0 0x55c1c1e63680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.299+0000 7effa1db1640 1 -- 172.21.15.60:0/2817847155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1c1e649d0 msgr2=0x55c1c1eedf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.299+0000 7effa1db1640 1 --2- 172.21.15.60:0/2817847155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1c1e649d0 0x55c1c1eedf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.299+0000 7effa1db1640 1 -- 172.21.15.60:0/2817847155 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1c1e5e120 con 0x55c1c1e7e770 2024-05-09T05:01:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.299+0000 7effa1db1640 1 --2- 172.21.15.60:0/2817847155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c1c1e7e770 0x55c1c1e61f70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7eff94025cc0 tx=0x7eff94051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=114b02d56222659b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.299+0000 7eff9b7fe640 1 -- 172.21.15.60:0/2817847155 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff94056720 con 0x55c1c1e7e770 2024-05-09T05:01:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.299+0000 7eff9b7fe640 1 -- 172.21.15.60:0/2817847155 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff940568c0 con 0x55c1c1e7e770 2024-05-09T05:01:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.299+0000 7eff9b7fe640 1 -- 172.21.15.60:0/2817847155 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff94056c70 con 0x55c1c1e7e770 2024-05-09T05:01:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.299+0000 7effa451af40 1 -- 172.21.15.60:0/2817847155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c1c1e7e770 msgr2=0x55c1c1e61f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.299+0000 7effa451af40 1 --2- 172.21.15.60:0/2817847155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c1c1e7e770 0x55c1c1e61f70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7eff94025cc0 tx=0x7eff94051000 comp rx=0 tx=0).stop 2024-05-09T05:01:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.299+0000 7effa451af40 1 -- 172.21.15.60:0/2817847155 shutdown_connections 2024-05-09T05:01:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.299+0000 7effa451af40 1 --2- 172.21.15.60:0/2817847155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c1c1e7e770 0x55c1c1e61f70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.299+0000 7effa451af40 1 --2- 172.21.15.60:0/2817847155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1c1e649d0 0x55c1c1eedf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.299+0000 7effa451af40 1 --2- 172.21.15.60:0/2817847155 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1c1e321f0 0x55c1c1e63680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.299+0000 7effa451af40 1 -- 172.21.15.60:0/2817847155 >> 172.21.15.60:0/2817847155 conn(0x55c1c1e34110 msgr2=0x55c1c1e361f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:21.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.300+0000 7effa451af40 1 -- 172.21.15.60:0/2817847155 shutdown_connections 2024-05-09T05:01:21.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.300+0000 7effa451af40 1 -- 172.21.15.60:0/2817847155 wait complete. 2024-05-09T05:01:21.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.300+0000 7effa451af40 1 Processor -- start 2024-05-09T05:01:21.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.300+0000 7effa451af40 1 -- start start 2024-05-09T05:01:21.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.301+0000 7effa451af40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1c1e321f0 0x55c1c1eea550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.301+0000 7effa451af40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1c1e649d0 0x55c1c1ef02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.301+0000 7effa451af40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c1c1e7e770 0x55c1c1ef0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.301+0000 7effa451af40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x55c1c1eee5c0 con 0x55c1c1e7e770 2024-05-09T05:01:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.301+0000 7effa451af40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c1c1f09e90 con 0x55c1c1e321f0 2024-05-09T05:01:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.301+0000 7effa451af40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x55c1c1f0a170 con 0x55c1c1e649d0 2024-05-09T05:01:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.301+0000 7effa1db1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c1c1e7e770 0x55c1c1ef0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.301+0000 7effa1db1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c1c1e7e770 0x55c1c1ef0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59562/0 (socket says 172.21.15.60:59562) 2024-05-09T05:01:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.301+0000 7effa1db1640 1 -- 172.21.15.60:0/4019355431 learned_addr learned my addr 172.21.15.60:0/4019355431 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.301+0000 7effa15b0640 1 --2- 172.21.15.60:0/4019355431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1c1e321f0 0x55c1c1eea550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.301+0000 7effa0daf640 1 --2- 172.21.15.60:0/4019355431 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1c1e649d0 0x55c1c1ef02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.301+0000 7effa1db1640 1 -- 172.21.15.60:0/4019355431 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1c1e649d0 msgr2=0x55c1c1ef02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.301+0000 7effa1db1640 1 --2- 172.21.15.60:0/4019355431 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x55c1c1e649d0 0x55c1c1ef02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.301+0000 7effa1db1640 1 -- 172.21.15.60:0/4019355431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1c1e321f0 msgr2=0x55c1c1eea550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.301+0000 7effa1db1640 1 --2- 172.21.15.60:0/4019355431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1c1e321f0 0x55c1c1eea550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.301+0000 7effa1db1640 1 -- 172.21.15.60:0/4019355431 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1c1eeb530 con 0x55c1c1e7e770 2024-05-09T05:01:21.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.302+0000 7effa1db1640 1 --2- 172.21.15.60:0/4019355431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c1c1e7e770 0x55c1c1ef0810 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7eff94056040 tx=0x7eff94002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:21.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.302+0000 7eff9a7fc640 1 -- 172.21.15.60:0/4019355431 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff94076070 con 0x55c1c1e7e770 2024-05-09T05:01:21.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.302+0000 7eff9a7fc640 1 -- 172.21.15.60:0/4019355431 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff94056c70 con 0x55c1c1e7e770 2024-05-09T05:01:21.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.302+0000 7effa451af40 1 -- 172.21.15.60:0/4019355431 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c1c1eeb810 con 0x55c1c1e7e770 2024-05-09T05:01:21.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.302+0000 7eff9a7fc640 1 -- 172.21.15.60:0/4019355431 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff94054440 con 0x55c1c1e7e770 2024-05-09T05:01:21.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.302+0000 7effa451af40 1 -- 172.21.15.60:0/4019355431 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1c1eebdd0 con 0x55c1c1e7e770 2024-05-09T05:01:21.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.303+0000 7eff9a7fc640 1 -- 172.21.15.60:0/4019355431 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7eff94079070 con 0x55c1c1e7e770 2024-05-09T05:01:21.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.304+0000 7eff9a7fc640 1 --2- 172.21.15.60:0/4019355431 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7eff84074cb0 0x7eff84077170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.304+0000 7eff9a7fc640 1 -- 172.21.15.60:0/4019355431 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6496+0+0 (secure 0 0 0) 0x7eff940dbdd0 con 0x55c1c1e7e770 2024-05-09T05:01:21.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.304+0000 7effa15b0640 1 --2- 172.21.15.60:0/4019355431 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7eff84074cb0 0x7eff84077170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.304+0000 7effa451af40 1 --2- 172.21.15.60:0/4019355431 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x55c1c1f0e090 0x55c1c1eecae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.304+0000 7effa451af40 1 -- 172.21.15.60:0/4019355431 --> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] -- osd_op(unknown.0.0:1 7.13 7:ca8fdbba:::100000001fb.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x55c1c1eed020 con 0x55c1c1f0e090 2024-05-09T05:01:21.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.304+0000 7effa0daf640 1 --2- 172.21.15.60:0/4019355431 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x55c1c1f0e090 0x55c1c1eecae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.305+0000 7effa15b0640 1 --2- 172.21.15.60:0/4019355431 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7eff84074cb0 0x7eff84077170 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7eff9c006850 tx=0x7eff9c0067e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:21.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.305+0000 7effa0daf640 1 --2- 172.21.15.60:0/4019355431 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x55c1c1f0e090 0x55c1c1eecae0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:21.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.305+0000 7effa0daf640 1 -- 172.21.15.60:0/4019355431 <== osd.0 v2:172.21.15.60:6800/1298086532 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (103) out=103b] v0'0 uv2 ondisk = 0) ==== 164+0+103 (crc 0 0 0) 0x7eff8c00e040 con 0x55c1c1f0e090 2024-05-09T05:01:21.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.305+0000 7effa451af40 1 -- 172.21.15.60:0/4019355431 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x55c1c1f0e090 msgr2=0x55c1c1eecae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:21.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.305+0000 7effa451af40 1 --2- 172.21.15.60:0/4019355431 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x55c1c1f0e090 0x55c1c1eecae0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.305+0000 7effa451af40 1 -- 172.21.15.60:0/4019355431 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7eff84074cb0 msgr2=0x7eff84077170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:21.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.305+0000 7effa451af40 1 --2- 172.21.15.60:0/4019355431 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7eff84074cb0 0x7eff84077170 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7eff9c006850 tx=0x7eff9c0067e0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.305+0000 7effa451af40 1 -- 172.21.15.60:0/4019355431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c1c1e7e770 msgr2=0x55c1c1ef0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:21.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.305+0000 7effa451af40 1 --2- 172.21.15.60:0/4019355431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c1c1e7e770 0x55c1c1ef0810 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7eff94056040 tx=0x7eff94002fb0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.305+0000 7effa1db1640 1 -- 172.21.15.60:0/4019355431 reap_dead start 2024-05-09T05:01:21.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.305+0000 7effa451af40 1 -- 172.21.15.60:0/4019355431 shutdown_connections 2024-05-09T05:01:21.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.305+0000 7effa451af40 1 -- 172.21.15.60:0/4019355431 >> 172.21.15.60:0/4019355431 conn(0x55c1c1e34110 msgr2=0x55c1c1e634f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:21.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.305+0000 7effa451af40 1 -- 172.21.15.60:0/4019355431 shutdown_connections 2024-05-09T05:01:21.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.305+0000 7effa451af40 1 -- 172.21.15.60:0/4019355431 wait complete. 2024-05-09T05:01:21.310 INFO:teuthology.orchestra.run.smithi060.stdout:a��alphaparent_c% 2024-05-09T05:01:21.311 DEBUG:teuthology.orchestra.run.smithi060:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-05-09T05:01:21.361 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:01:21.361 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628283, 2024-05-09T05:01:21.361 INFO:teuthology.orchestra.run.smithi060.stdout: "ancestors": [ 2024-05-09T05:01:21.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:21.361 INFO:teuthology.orchestra.run.smithi060.stdout: "dirino": 1099511628285, 2024-05-09T05:01:21.361 INFO:teuthology.orchestra.run.smithi060.stdout: "dname": "alpha", 2024-05-09T05:01:21.362 INFO:teuthology.orchestra.run.smithi060.stdout: "version": 8 2024-05-09T05:01:21.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:21.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:21.362 INFO:teuthology.orchestra.run.smithi060.stdout: "dirino": 1, 2024-05-09T05:01:21.362 INFO:teuthology.orchestra.run.smithi060.stdout: "dname": "parent_c", 2024-05-09T05:01:21.362 INFO:teuthology.orchestra.run.smithi060.stdout: "version": 37 2024-05-09T05:01:21.362 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:21.362 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-05-09T05:01:21.362 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 7, 2024-05-09T05:01:21.362 INFO:teuthology.orchestra.run.smithi060.stdout: "old_pools": [ 2024-05-09T05:01:21.362 INFO:teuthology.orchestra.run.smithi060.stdout: 6 2024-05-09T05:01:21.362 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-05-09T05:01:21.362 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:01:21.363 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:21.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:21.364 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.object_size -v 8388608 ./parent_c/alpha) 2024-05-09T05:01:21.424 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:21.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.568+0000 7faa6aa3e640 1 Processor -- start 2024-05-09T05:01:21.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.568+0000 7faa6aa3e640 1 -- start start 2024-05-09T05:01:21.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.569+0000 7faa6aa3e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa640d1050 0x7faa640ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.569+0000 7faa6aa3e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faa640d1430 0x7faa640d18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.569+0000 7faa6aa3e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa640af370 con 0x7faa64096eb0 2024-05-09T05:01:21.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.569+0000 7faa6aa3e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7faa64096a00 con 0x7faa640d1430 2024-05-09T05:01:21.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.569+0000 7faa6aa3e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7faa640d2dd0 con 0x7faa640d1050 2024-05-09T05:01:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.569+0000 7faa68fdb640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faa640d1430 0x7faa640d18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.569+0000 7faa637fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7faa64096eb0 0x7faa6409b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42080/0 (socket says 172.21.15.60:42080) 2024-05-09T05:01:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.569+0000 7faa637fe640 1 -- 172.21.15.60:0/690887773 learned_addr learned my addr 172.21.15.60:0/690887773 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.569+0000 7faa63fff640 1 --2- 172.21.15.60:0/690887773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa640d1050 0x7faa640ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.569+0000 7faa68fdb640 1 -- 172.21.15.60:0/690887773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa640d1050 msgr2=0x7faa640ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.569+0000 7faa68fdb640 1 --2- 172.21.15.60:0/690887773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa640d1050 0x7faa640ae9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.569+0000 7faa68fdb640 1 -- 172.21.15.60:0/690887773 >> v1:172.21.15.60:6789/0 conn(0x7faa64096eb0 legacy=0x7faa6409b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.569+0000 7faa68fdb640 1 -- 172.21.15.60:0/690887773 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa640d30b0 con 0x7faa640d1430 2024-05-09T05:01:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.570+0000 7faa68fdb640 1 --2- 172.21.15.60:0/690887773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa640d1430 0x7faa640d18e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7faa5c009d90 tx=0x7faa5c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=804d78ecbb484903 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:21.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.570+0000 7faa62ffd640 1 -- 172.21.15.60:0/690887773 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c007740 con 0x7faa640d1430 2024-05-09T05:01:21.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.570+0000 7faa62ffd640 1 -- 172.21.15.60:0/690887773 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa5c0078e0 con 0x7faa640d1430 2024-05-09T05:01:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.570+0000 7faa62ffd640 1 -- 172.21.15.60:0/690887773 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c007cb0 con 0x7faa640d1430 2024-05-09T05:01:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.571+0000 7faa6aa3e640 1 -- 172.21.15.60:0/690887773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa640d1430 msgr2=0x7faa640d18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.571+0000 7faa6aa3e640 1 --2- 172.21.15.60:0/690887773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa640d1430 0x7faa640d18e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7faa5c009d90 tx=0x7faa5c02a660 comp rx=0 tx=0).stop 2024-05-09T05:01:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.571+0000 7faa6aa3e640 1 -- 172.21.15.60:0/690887773 shutdown_connections 2024-05-09T05:01:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.571+0000 7faa6aa3e640 1 --2- 172.21.15.60:0/690887773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa640d1430 0x7faa640d18e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.571+0000 7faa6aa3e640 1 --2- 172.21.15.60:0/690887773 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa640d1050 0x7faa640ae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.571+0000 7faa6aa3e640 1 -- 172.21.15.60:0/690887773 >> 172.21.15.60:0/690887773 conn(0x7faa640993f0 msgr2=0x7faa64099800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.571+0000 7faa6aa3e640 1 -- 172.21.15.60:0/690887773 shutdown_connections 2024-05-09T05:01:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.571+0000 7faa6aa3e640 1 -- 172.21.15.60:0/690887773 wait complete. 2024-05-09T05:01:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.572+0000 7faa6aa3e640 1 Processor -- start 2024-05-09T05:01:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.572+0000 7faa6aa3e640 1 -- start start 2024-05-09T05:01:21.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.573+0000 7faa6aa3e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa64096eb0 0x7faa641e6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.573+0000 7faa6aa3e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa640d1050 0x7faa641e92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.573+0000 7faa6aa3e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa640d1430 0x7faa640b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.573+0000 7faa6aa3e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7faa6409c1e0 con 0x7faa640d1430 2024-05-09T05:01:21.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.573+0000 7faa6aa3e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7faa6409c060 con 0x7faa64096eb0 2024-05-09T05:01:21.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.573+0000 7faa6aa3e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7faa640d2790 con 0x7faa640d1050 2024-05-09T05:01:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.573+0000 7faa68fdb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa640d1430 0x7faa640b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.573+0000 7faa68fdb640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa640d1430 0x7faa640b1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59590/0 (socket says 172.21.15.60:59590) 2024-05-09T05:01:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.573+0000 7faa68fdb640 1 -- 172.21.15.60:0/3355805387 learned_addr learned my addr 172.21.15.60:0/3355805387 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.573+0000 7faa68fdb640 1 -- 172.21.15.60:0/3355805387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa640d1050 msgr2=0x7faa641e92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.573+0000 7faa63fff640 1 --2- 172.21.15.60:0/3355805387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa64096eb0 0x7faa641e6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.573+0000 7faa637fe640 1 --2- 172.21.15.60:0/3355805387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa640d1050 0x7faa641e92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.573+0000 7faa68fdb640 1 --2- 172.21.15.60:0/3355805387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa640d1050 0x7faa641e92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.573+0000 7faa68fdb640 1 -- 172.21.15.60:0/3355805387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa64096eb0 msgr2=0x7faa641e6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.573+0000 7faa68fdb640 1 --2- 172.21.15.60:0/3355805387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa64096eb0 0x7faa641e6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.573+0000 7faa68fdb640 1 -- 172.21.15.60:0/3355805387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa6406d360 con 0x7faa640d1430 2024-05-09T05:01:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.574+0000 7faa68fdb640 1 --2- 172.21.15.60:0/3355805387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa640d1430 0x7faa640b1250 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7faa5c02ac50 tx=0x7faa5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.574+0000 7faa60ff9640 1 -- 172.21.15.60:0/3355805387 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c040070 con 0x7faa640d1430 2024-05-09T05:01:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.574+0000 7faa60ff9640 1 -- 172.21.15.60:0/3355805387 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa5c031bc0 con 0x7faa640d1430 2024-05-09T05:01:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.574+0000 7faa6aa3e640 1 -- 172.21.15.60:0/3355805387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa6406c7e0 con 0x7faa640d1430 2024-05-09T05:01:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.574+0000 7faa60ff9640 1 -- 172.21.15.60:0/3355805387 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c045460 con 0x7faa640d1430 2024-05-09T05:01:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.574+0000 7faa6aa3e640 1 -- 172.21.15.60:0/3355805387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa6406cda0 con 0x7faa640d1430 2024-05-09T05:01:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.575+0000 7faa6aa3e640 1 -- 172.21.15.60:0/3355805387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa64075540 con 0x7faa640d1430 2024-05-09T05:01:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.575+0000 7faa60ff9640 1 -- 172.21.15.60:0/3355805387 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faa5c039040 con 0x7faa640d1430 2024-05-09T05:01:21.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.576+0000 7faa60ff9640 1 --2- 172.21.15.60:0/3355805387 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7faa50076300 0x7faa500787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:21.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.576+0000 7faa60ff9640 1 -- 172.21.15.60:0/3355805387 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6496+0+0 (secure 0 0 0) 0x7faa5c0b9240 con 0x7faa640d1430 2024-05-09T05:01:21.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.576+0000 7faa63fff640 1 --2- 172.21.15.60:0/3355805387 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7faa50076300 0x7faa500787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:21.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.577+0000 7faa63fff640 1 --2- 172.21.15.60:0/3355805387 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7faa50076300 0x7faa500787c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faa54006fd0 tx=0x7faa54008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:21.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.580+0000 7faa60ff9640 1 -- 172.21.15.60:0/3355805387 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7faa5c082810 con 0x7faa640d1430 2024-05-09T05:01:21.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.810+0000 7faa6aa3e640 1 -- 172.21.15.60:0/3355805387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa64051040 con 0x7faa640d1430 2024-05-09T05:01:21.811 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.811+0000 7faa60ff9640 1 -- 172.21.15.60:0/3355805387 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5183 (secure 0 0 0) 0x7faa5c082110 con 0x7faa640d1430 2024-05-09T05:01:21.812 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:21.812 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4547,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2372252858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2372252858},{"type":"v1","addr":"172.21.15.114:6836","nonce":2372252858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1761282610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1761282610},{"type":"v1","addr":"172.21.15.114:6837","nonce":1761282610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4765,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/300408012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":300408012},{"type":"v1","addr":"172.21.15.60:6835","nonce":300408012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:04.698975+0000","modified":"2024-05-09T05:01:18.144520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4893},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4893":{"gid":4893,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/399160914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":399160914},{"type":"v1","addr":"172.21.15.60:6834","nonce":399160914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4893,"qdb_cluster":[4893]},"id":2}]} 2024-05-09T05:01:21.812 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 18 2024-05-09T05:01:21.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.813+0000 7faa6aa3e640 1 -- 172.21.15.60:0/3355805387 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7faa50076300 msgr2=0x7faa500787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:21.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.813+0000 7faa6aa3e640 1 --2- 172.21.15.60:0/3355805387 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7faa50076300 0x7faa500787c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faa54006fd0 tx=0x7faa54008040 comp rx=0 tx=0).stop 2024-05-09T05:01:21.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.813+0000 7faa6aa3e640 1 -- 172.21.15.60:0/3355805387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa640d1430 msgr2=0x7faa640b1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:21.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.813+0000 7faa6aa3e640 1 --2- 172.21.15.60:0/3355805387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa640d1430 0x7faa640b1250 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7faa5c02ac50 tx=0x7faa5c004a40 comp rx=0 tx=0).stop 2024-05-09T05:01:21.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.813+0000 7faa6aa3e640 1 -- 172.21.15.60:0/3355805387 shutdown_connections 2024-05-09T05:01:21.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.813+0000 7faa6aa3e640 1 --2- 172.21.15.60:0/3355805387 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7faa50076300 0x7faa500787c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.813+0000 7faa6aa3e640 1 --2- 172.21.15.60:0/3355805387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa640d1430 0x7faa640b1250 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.813+0000 7faa6aa3e640 1 --2- 172.21.15.60:0/3355805387 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa640d1050 0x7faa641e92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.813+0000 7faa6aa3e640 1 --2- 172.21.15.60:0/3355805387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa64096eb0 0x7faa641e6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:21.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.813+0000 7faa6aa3e640 1 -- 172.21.15.60:0/3355805387 >> 172.21.15.60:0/3355805387 conn(0x7faa640993f0 msgr2=0x7faa640afaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:21.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.813+0000 7faa6aa3e640 1 -- 172.21.15.60:0/3355805387 shutdown_connections 2024-05-09T05:01:21.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.813+0000 7faa6aa3e640 1 -- 172.21.15.60:0/3355805387 wait complete. 2024-05-09T05:01:21.825 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-05-09T05:01:21.953 INFO:teuthology.orchestra.run.smithi060.stdout:{"message":"","return_code":0} 2024-05-09T05:01:21.953 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-05-09T05:01:21.953 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool data_new getxattr 100000001fb.00000000 layout 2024-05-09T05:01:21.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.997+0000 7f617db35f40 1 Processor -- start 2024-05-09T05:01:21.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.997+0000 7f617db35f40 1 -- start start 2024-05-09T05:01:22.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.998+0000 7f617db35f40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x555eab440700 0x555eab4ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.998+0000 7f617db35f40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555eab45a8d0 0x555eab43c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.998+0000 7f617db35f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555eab4cfcd0 0x555eab4e4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.998+0000 7f617db35f40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x555eab3b8f50 con 0x555eab45a8d0 2024-05-09T05:01:22.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.998+0000 7f617db35f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555eab3bcf90 con 0x555eab4cfcd0 2024-05-09T05:01:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.998+0000 7f617db35f40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x555eab4410b0 con 0x555eab440700 2024-05-09T05:01:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.998+0000 7f6173fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555eab45a8d0 0x555eab43c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.998+0000 7f6173fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555eab45a8d0 0x555eab43c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59610/0 (socket says 172.21.15.60:59610) 2024-05-09T05:01:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.998+0000 7f6173fff640 1 -- 172.21.15.60:0/4249571057 learned_addr learned my addr 172.21.15.60:0/4249571057 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.998+0000 7f6173fff640 1 -- 172.21.15.60:0/4249571057 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x555eab440700 msgr2=0x555eab4ceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.998+0000 7f6173fff640 1 --2- 172.21.15.60:0/4249571057 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x555eab440700 0x555eab4ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.998+0000 7f6173fff640 1 -- 172.21.15.60:0/4249571057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555eab4cfcd0 msgr2=0x555eab4e4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.998+0000 7f6173fff640 1 --2- 172.21.15.60:0/4249571057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555eab4cfcd0 0x555eab4e4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.998+0000 7f6173fff640 1 -- 172.21.15.60:0/4249571057 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555eab43cef0 con 0x555eab45a8d0 2024-05-09T05:01:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.999+0000 7f6173fff640 1 --2- 172.21.15.60:0/4249571057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555eab45a8d0 0x555eab43c1a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f616400a430 tx=0x7f616400a8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d695e094be085 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.999+0000 7f617a62d640 1 -- 172.21.15.60:0/4249571057 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616400b920 con 0x555eab45a8d0 2024-05-09T05:01:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.999+0000 7f617a62d640 1 -- 172.21.15.60:0/4249571057 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f616400f030 con 0x555eab45a8d0 2024-05-09T05:01:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.999+0000 7f617a62d640 1 -- 172.21.15.60:0/4249571057 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616400bd20 con 0x555eab45a8d0 2024-05-09T05:01:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.999+0000 7f617db35f40 1 -- 172.21.15.60:0/4249571057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555eab45a8d0 msgr2=0x555eab43c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:22.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.999+0000 7f617db35f40 1 --2- 172.21.15.60:0/4249571057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555eab45a8d0 0x555eab43c1a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f616400a430 tx=0x7f616400a8f0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.999+0000 7f617db35f40 1 -- 172.21.15.60:0/4249571057 shutdown_connections 2024-05-09T05:01:22.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.999+0000 7f617db35f40 1 --2- 172.21.15.60:0/4249571057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555eab4cfcd0 0x555eab4e4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.999+0000 7f617db35f40 1 --2- 172.21.15.60:0/4249571057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555eab45a8d0 0x555eab43c1a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.999+0000 7f617db35f40 1 --2- 172.21.15.60:0/4249571057 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x555eab440700 0x555eab4ceda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.999+0000 7f617db35f40 1 -- 172.21.15.60:0/4249571057 >> 172.21.15.60:0/4249571057 conn(0x555eab410890 msgr2=0x555eab459170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:22.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.999+0000 7f617db35f40 1 -- 172.21.15.60:0/4249571057 shutdown_connections 2024-05-09T05:01:22.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:21.999+0000 7f617db35f40 1 -- 172.21.15.60:0/4249571057 wait complete. 2024-05-09T05:01:22.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.000+0000 7f617db35f40 1 Processor -- start 2024-05-09T05:01:22.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.000+0000 7f617db35f40 1 -- start start 2024-05-09T05:01:22.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f617db35f40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555eab440700 0x555eab4e3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f617db35f40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555eab45a8d0 0x555eab4e4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f617db35f40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x555eab4cfcd0 0x555eab4e1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f617db35f40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x555eab4c61c0 con 0x555eab45a8d0 2024-05-09T05:01:22.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f617db35f40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555eab4e5650 con 0x555eab440700 2024-05-09T05:01:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f617db35f40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x555eab410c70 con 0x555eab4cfcd0 2024-05-09T05:01:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f6173fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555eab45a8d0 0x555eab4e4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f6173fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555eab45a8d0 0x555eab4e4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59620/0 (socket says 172.21.15.60:59620) 2024-05-09T05:01:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f6173fff640 1 -- 172.21.15.60:0/3014024164 learned_addr learned my addr 172.21.15.60:0/3014024164 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f617b62f640 1 --2- 172.21.15.60:0/3014024164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555eab440700 0x555eab4e3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f617be30640 1 --2- 172.21.15.60:0/3014024164 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x555eab4cfcd0 0x555eab4e1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f6173fff640 1 -- 172.21.15.60:0/3014024164 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x555eab4cfcd0 msgr2=0x555eab4e1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f6173fff640 1 --2- 172.21.15.60:0/3014024164 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x555eab4cfcd0 0x555eab4e1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f6173fff640 1 -- 172.21.15.60:0/3014024164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555eab440700 msgr2=0x555eab4e3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f6173fff640 1 --2- 172.21.15.60:0/3014024164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555eab440700 0x555eab4e3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f6173fff640 1 -- 172.21.15.60:0/3014024164 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555eab4e4530 con 0x555eab45a8d0 2024-05-09T05:01:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f6173fff640 1 --2- 172.21.15.60:0/3014024164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555eab45a8d0 0x555eab4e4b10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f616400f000 tx=0x7f6164002f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f617962b640 1 -- 172.21.15.60:0/3014024164 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6164024070 con 0x555eab45a8d0 2024-05-09T05:01:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f617962b640 1 -- 172.21.15.60:0/3014024164 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6164014420 con 0x555eab45a8d0 2024-05-09T05:01:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f617db35f40 1 -- 172.21.15.60:0/3014024164 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555eab4e3c90 con 0x555eab45a8d0 2024-05-09T05:01:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.001+0000 7f617962b640 1 -- 172.21.15.60:0/3014024164 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61640147f0 con 0x555eab45a8d0 2024-05-09T05:01:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.002+0000 7f617db35f40 1 -- 172.21.15.60:0/3014024164 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555eab4e4250 con 0x555eab45a8d0 2024-05-09T05:01:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.003+0000 7f617962b640 1 -- 172.21.15.60:0/3014024164 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6164027070 con 0x555eab45a8d0 2024-05-09T05:01:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.003+0000 7f617962b640 1 --2- 172.21.15.60:0/3014024164 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f615c074cb0 0x7f615c077170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.004+0000 7f617962b640 1 -- 172.21.15.60:0/3014024164 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6496+0+0 (secure 0 0 0) 0x7f6164097540 con 0x555eab45a8d0 2024-05-09T05:01:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.004+0000 7f617b62f640 1 --2- 172.21.15.60:0/3014024164 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f615c074cb0 0x7f615c077170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.004+0000 7f617db35f40 1 --2- 172.21.15.60:0/3014024164 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x555eab4e5050 0x555eab4c4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.004+0000 7f617db35f40 1 -- 172.21.15.60:0/3014024164 --> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] -- osd_op(unknown.0.0:1 7.13 7:ca8fdbba:::100000001fb.00000000:head [getxattr layout in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x555eab4e7270 con 0x555eab4e5050 2024-05-09T05:01:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.004+0000 7f617be30640 1 --2- 172.21.15.60:0/3014024164 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x555eab4e5050 0x555eab4c4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.004+0000 7f617b62f640 1 --2- 172.21.15.60:0/3014024164 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f615c074cb0 0x7f615c077170 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f616c000f60 tx=0x7f616c001530 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.004+0000 7f617be30640 1 --2- 172.21.15.60:0/3014024164 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x555eab4e5050 0x555eab4c4de0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:22.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.004+0000 7f617be30640 1 -- 172.21.15.60:0/3014024164 <== osd.0 v2:172.21.15.60:6800/1298086532 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (30) out=30b] v0'0 uv3 ondisk = 0) ==== 164+0+30 (crc 0 0 0) 0x7f617404e040 con 0x555eab4e5050 2024-05-09T05:01:22.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.005+0000 7f617db35f40 1 -- 172.21.15.60:0/3014024164 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x555eab4e5050 msgr2=0x555eab4c4de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:22.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.005+0000 7f617db35f40 1 --2- 172.21.15.60:0/3014024164 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x555eab4e5050 0x555eab4c4de0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.005+0000 7f617db35f40 1 -- 172.21.15.60:0/3014024164 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f615c074cb0 msgr2=0x7f615c077170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:22.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.005+0000 7f617db35f40 1 --2- 172.21.15.60:0/3014024164 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f615c074cb0 0x7f615c077170 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f616c000f60 tx=0x7f616c001530 comp rx=0 tx=0).stop 2024-05-09T05:01:22.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.005+0000 7f617db35f40 1 -- 172.21.15.60:0/3014024164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555eab45a8d0 msgr2=0x555eab4e4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:22.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.005+0000 7f617db35f40 1 --2- 172.21.15.60:0/3014024164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555eab45a8d0 0x555eab4e4b10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f616400f000 tx=0x7f6164002f70 comp rx=0 tx=0).stop 2024-05-09T05:01:22.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.005+0000 7f617be30640 1 -- 172.21.15.60:0/3014024164 reap_dead start 2024-05-09T05:01:22.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.005+0000 7f617db35f40 1 -- 172.21.15.60:0/3014024164 shutdown_connections 2024-05-09T05:01:22.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.005+0000 7f617db35f40 1 -- 172.21.15.60:0/3014024164 >> 172.21.15.60:0/3014024164 conn(0x555eab410890 msgr2=0x555eab459170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:22.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.005+0000 7f617db35f40 1 -- 172.21.15.60:0/3014024164 shutdown_connections 2024-05-09T05:01:22.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.005+0000 7f617db35f40 1 -- 172.21.15.60:0/3014024164 wait complete. 2024-05-09T05:01:22.010 INFO:teuthology.orchestra.run.smithi060.stdout:@� 2024-05-09T05:01:22.010 DEBUG:teuthology.orchestra.run.smithi060:> ceph-dencoder type file_layout_t import - decode dump_json 2024-05-09T05:01:22.058 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:01:22.059 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_unit": 4194304, 2024-05-09T05:01:22.059 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_count": 1, 2024-05-09T05:01:22.059 INFO:teuthology.orchestra.run.smithi060.stdout: "object_size": 8388608, 2024-05-09T05:01:22.059 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_id": 7, 2024-05-09T05:01:22.059 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_ns": "" 2024-05-09T05:01:22.059 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:01:22.059 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:22.061 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001fb.00000000 layout 2024-05-09T05:01:22.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.145+0000 7f0c9c71cf40 1 Processor -- start 2024-05-09T05:01:22.146 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.145+0000 7f0c9c71cf40 1 -- start start 2024-05-09T05:01:22.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.146+0000 7f0c9c71cf40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563bf7da6c40 0x563bf7e32b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.146+0000 7f0c9c71cf40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bf7dbf6b0 0x563bf7e33050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.146+0000 7f0c9c71cf40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563bf7e35cd0 0x563bf7dbb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.146+0000 7f0c9c71cf40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x563bf7da70b0 con 0x563bf7e35cd0 2024-05-09T05:01:22.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.146+0000 7f0c9c71cf40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563bf7d1ef50 con 0x563bf7dbf6b0 2024-05-09T05:01:22.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.146+0000 7f0c9c71cf40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x563bf7dbe3b0 con 0x563bf7da6c40 2024-05-09T05:01:22.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.146+0000 7f0c99fb3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563bf7e35cd0 0x563bf7dbb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.146+0000 7f0c99fb3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563bf7e35cd0 0x563bf7dbb8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59634/0 (socket says 172.21.15.60:59634) 2024-05-09T05:01:22.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.146+0000 7f0c99fb3640 1 -- 172.21.15.60:0/695536888 learned_addr learned my addr 172.21.15.60:0/695536888 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:22.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.146+0000 7f0c98fb1640 1 --2- 172.21.15.60:0/695536888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bf7dbf6b0 0x563bf7e33050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.146+0000 7f0c997b2640 1 --2- 172.21.15.60:0/695536888 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563bf7da6c40 0x563bf7e32b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.146+0000 7f0c99fb3640 1 -- 172.21.15.60:0/695536888 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563bf7da6c40 msgr2=0x563bf7e32b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:22.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.146+0000 7f0c99fb3640 1 --2- 172.21.15.60:0/695536888 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563bf7da6c40 0x563bf7e32b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.146+0000 7f0c99fb3640 1 -- 172.21.15.60:0/695536888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bf7dbf6b0 msgr2=0x563bf7e33050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:22.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.146+0000 7f0c99fb3640 1 --2- 172.21.15.60:0/695536888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bf7dbf6b0 0x563bf7e33050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.146+0000 7f0c99fb3640 1 -- 172.21.15.60:0/695536888 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bf7da26a0 con 0x563bf7e35cd0 2024-05-09T05:01:22.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.146+0000 7f0c99fb3640 1 --2- 172.21.15.60:0/695536888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563bf7e35cd0 0x563bf7dbb8d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c001690 tx=0x7f0c8c001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a80e1cd54cd80e24 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:22.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.147+0000 7f0c937fe640 1 -- 172.21.15.60:0/695536888 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c017070 con 0x563bf7e35cd0 2024-05-09T05:01:22.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.147+0000 7f0c937fe640 1 -- 172.21.15.60:0/695536888 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c8c00f710 con 0x563bf7e35cd0 2024-05-09T05:01:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.147+0000 7f0c937fe640 1 -- 172.21.15.60:0/695536888 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c00fae0 con 0x563bf7e35cd0 2024-05-09T05:01:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.147+0000 7f0c9c71cf40 1 -- 172.21.15.60:0/695536888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563bf7e35cd0 msgr2=0x563bf7dbb8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.147+0000 7f0c9c71cf40 1 --2- 172.21.15.60:0/695536888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563bf7e35cd0 0x563bf7dbb8d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c001690 tx=0x7f0c8c001b50 comp rx=0 tx=0).stop 2024-05-09T05:01:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.147+0000 7f0c9c71cf40 1 -- 172.21.15.60:0/695536888 shutdown_connections 2024-05-09T05:01:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.147+0000 7f0c9c71cf40 1 --2- 172.21.15.60:0/695536888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563bf7e35cd0 0x563bf7dbb8d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.147+0000 7f0c9c71cf40 1 --2- 172.21.15.60:0/695536888 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bf7dbf6b0 0x563bf7e33050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.147+0000 7f0c9c71cf40 1 --2- 172.21.15.60:0/695536888 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563bf7da6c40 0x563bf7e32b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.147+0000 7f0c9c71cf40 1 -- 172.21.15.60:0/695536888 >> 172.21.15.60:0/695536888 conn(0x563bf7d76890 msgr2=0x563bf7dc0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.147+0000 7f0c9c71cf40 1 -- 172.21.15.60:0/695536888 shutdown_connections 2024-05-09T05:01:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.147+0000 7f0c9c71cf40 1 -- 172.21.15.60:0/695536888 wait complete. 2024-05-09T05:01:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.148+0000 7f0c9c71cf40 1 Processor -- start 2024-05-09T05:01:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.148+0000 7f0c9c71cf40 1 -- start start 2024-05-09T05:01:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.148+0000 7f0c9c71cf40 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563bf7da6c40 0x563bf7ee6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.148+0000 7f0c9c71cf40 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563bf7dbf6b0 0x563bf7ee7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.148+0000 7f0c9c71cf40 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bf7e35cd0 0x563bf7e33a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.148+0000 7f0c9c71cf40 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x563bf7da2960 con 0x563bf7dbf6b0 2024-05-09T05:01:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.148+0000 7f0c9c71cf40 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563bf7dbd390 con 0x563bf7e35cd0 2024-05-09T05:01:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.148+0000 7f0c9c71cf40 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x563bf7e4b540 con 0x563bf7da6c40 2024-05-09T05:01:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.148+0000 7f0c98fb1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563bf7dbf6b0 0x563bf7ee7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.149+0000 7f0c98fb1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563bf7dbf6b0 0x563bf7ee7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59640/0 (socket says 172.21.15.60:59640) 2024-05-09T05:01:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.149+0000 7f0c98fb1640 1 -- 172.21.15.60:0/1765529657 learned_addr learned my addr 172.21.15.60:0/1765529657 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.149+0000 7f0c997b2640 1 --2- 172.21.15.60:0/1765529657 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563bf7da6c40 0x563bf7ee6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.149+0000 7f0c99fb3640 1 --2- 172.21.15.60:0/1765529657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bf7e35cd0 0x563bf7e33a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.149+0000 7f0c98fb1640 1 -- 172.21.15.60:0/1765529657 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563bf7da6c40 msgr2=0x563bf7ee6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.149+0000 7f0c98fb1640 1 --2- 172.21.15.60:0/1765529657 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x563bf7da6c40 0x563bf7ee6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.149+0000 7f0c98fb1640 1 -- 172.21.15.60:0/1765529657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bf7e35cd0 msgr2=0x563bf7e33a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.149+0000 7f0c98fb1640 1 --2- 172.21.15.60:0/1765529657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563bf7e35cd0 0x563bf7e33a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.149+0000 7f0c98fb1640 1 -- 172.21.15.60:0/1765529657 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bf7ee82b0 con 0x563bf7dbf6b0 2024-05-09T05:01:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.149+0000 7f0c98fb1640 1 --2- 172.21.15.60:0/1765529657 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563bf7dbf6b0 0x563bf7ee7060 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0c84055040 tx=0x7f0c84002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.149+0000 7f0c927fc640 1 -- 172.21.15.60:0/1765529657 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c8406c070 con 0x563bf7dbf6b0 2024-05-09T05:01:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.149+0000 7f0c927fc640 1 -- 172.21.15.60:0/1765529657 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c84053970 con 0x563bf7dbf6b0 2024-05-09T05:01:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.149+0000 7f0c9c71cf40 1 -- 172.21.15.60:0/1765529657 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563bf7ee7730 con 0x563bf7dbf6b0 2024-05-09T05:01:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.149+0000 7f0c927fc640 1 -- 172.21.15.60:0/1765529657 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c84053d60 con 0x563bf7dbf6b0 2024-05-09T05:01:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.149+0000 7f0c9c71cf40 1 -- 172.21.15.60:0/1765529657 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bf7ee7cf0 con 0x563bf7dbf6b0 2024-05-09T05:01:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.151+0000 7f0c927fc640 1 -- 172.21.15.60:0/1765529657 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0c8406f070 con 0x563bf7dbf6b0 2024-05-09T05:01:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.151+0000 7f0c927fc640 1 --2- 172.21.15.60:0/1765529657 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0c7c074cb0 0x7f0c7c077170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.151+0000 7f0c927fc640 1 -- 172.21.15.60:0/1765529657 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6496+0+0 (secure 0 0 0) 0x7f0c840da650 con 0x563bf7dbf6b0 2024-05-09T05:01:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.151+0000 7f0c997b2640 1 --2- 172.21.15.60:0/1765529657 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0c7c074cb0 0x7f0c7c077170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.151+0000 7f0c9c71cf40 1 --2- 172.21.15.60:0/1765529657 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x563bf7e34450 0x563bf7ef0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.151+0000 7f0c9c71cf40 1 -- 172.21.15.60:0/1765529657 --> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] -- osd_op(unknown.0.0:1 6.13 6:ca8fdbba:::100000001fb.00000000:head [getxattr layout in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x563bf7ef0ac0 con 0x563bf7e34450 2024-05-09T05:01:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.152+0000 7f0c99fb3640 1 --2- 172.21.15.60:0/1765529657 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x563bf7e34450 0x563bf7ef0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.152+0000 7f0c99fb3640 1 --2- 172.21.15.60:0/1765529657 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x563bf7e34450 0x563bf7ef0580 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.152+0000 7f0c997b2640 1 --2- 172.21.15.60:0/1765529657 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0c7c074cb0 0x7f0c7c077170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0c94009700 tx=0x7f0c94005aa0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.152+0000 7f0c99fb3640 1 -- 172.21.15.60:0/1765529657 <== osd.3 v2:172.21.15.60:6824/1069806621 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (30) out=30b] v0'0 uv3 ondisk = 0) ==== 164+0+30 (crc 0 0 0) 0x7f0c8c003030 con 0x563bf7e34450 2024-05-09T05:01:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.152+0000 7f0c9c71cf40 1 -- 172.21.15.60:0/1765529657 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x563bf7e34450 msgr2=0x563bf7ef0580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.152+0000 7f0c9c71cf40 1 --2- 172.21.15.60:0/1765529657 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x563bf7e34450 0x563bf7ef0580 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.152+0000 7f0c9c71cf40 1 -- 172.21.15.60:0/1765529657 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0c7c074cb0 msgr2=0x7f0c7c077170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.152+0000 7f0c9c71cf40 1 --2- 172.21.15.60:0/1765529657 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0c7c074cb0 0x7f0c7c077170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0c94009700 tx=0x7f0c94005aa0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.152+0000 7f0c9c71cf40 1 -- 172.21.15.60:0/1765529657 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563bf7dbf6b0 msgr2=0x563bf7ee7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:22.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.152+0000 7f0c9c71cf40 1 --2- 172.21.15.60:0/1765529657 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563bf7dbf6b0 0x563bf7ee7060 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0c84055040 tx=0x7f0c84002fb0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.152+0000 7f0c99fb3640 1 -- 172.21.15.60:0/1765529657 reap_dead start 2024-05-09T05:01:22.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.152+0000 7f0c9c71cf40 1 -- 172.21.15.60:0/1765529657 shutdown_connections 2024-05-09T05:01:22.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.152+0000 7f0c9c71cf40 1 -- 172.21.15.60:0/1765529657 >> 172.21.15.60:0/1765529657 conn(0x563bf7d76890 msgr2=0x563bf7e32b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:22.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.152+0000 7f0c9c71cf40 1 -- 172.21.15.60:0/1765529657 shutdown_connections 2024-05-09T05:01:22.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.152+0000 7f0c9c71cf40 1 -- 172.21.15.60:0/1765529657 wait complete. 2024-05-09T05:01:22.158 INFO:teuthology.orchestra.run.smithi060.stdout:@@ 2024-05-09T05:01:22.158 DEBUG:teuthology.orchestra.run.smithi060:> ceph-dencoder type file_layout_t import - decode dump_json 2024-05-09T05:01:22.247 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:01:22.247 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_unit": 4194304, 2024-05-09T05:01:22.248 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_count": 1, 2024-05-09T05:01:22.248 INFO:teuthology.orchestra.run.smithi060.stdout: "object_size": 4194304, 2024-05-09T05:01:22.248 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_id": 6, 2024-05-09T05:01:22.248 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_ns": "" 2024-05-09T05:01:22.248 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:01:22.248 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:22.250 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T05:01:22.250 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T05:01:22.330 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T05:01:22.379 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T05:01:22.379 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T05:01:22.379 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:01:22.379 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:01:22.397 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:01:22.397 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:01:22.454 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T05:01:22.596 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:01:22.599 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:01:22.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.599+0000 7ff0ff07c640 1 Processor -- start 2024-05-09T05:01:22.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.599+0000 7ff0ff07c640 1 -- start start 2024-05-09T05:01:22.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.600+0000 7ff0ff07c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff0f8096c90 0x7ff0f80aea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.600+0000 7ff0ff07c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0f80b1110 0x7ff0f80b2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.600+0000 7ff0ff07c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0f80b1650 con 0x7ff0f80aef70 2024-05-09T05:01:22.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.600+0000 7ff0ff07c640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ff0f8160a90 con 0x7ff0f8096c90 2024-05-09T05:01:22.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.600+0000 7ff0ff07c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff0f80b1fb0 con 0x7ff0f80b1110 2024-05-09T05:01:22.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.600+0000 7ff0f7fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff0f80aef70 0x7ff0f80b0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42106/0 (socket says 172.21.15.60:42106) 2024-05-09T05:01:22.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.600+0000 7ff0f7fff640 1 -- 172.21.15.60:0/655264290 learned_addr learned my addr 172.21.15.60:0/655264290 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:22.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.600+0000 7ff0fce18640 1 --2- 172.21.15.60:0/655264290 >> v2:172.21.15.114:3300/0 conn(0x7ff0f8096c90 0x7ff0f80aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.600+0000 7ff0fd619640 1 --2- 172.21.15.60:0/655264290 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0f80b1110 0x7ff0f80b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.600+0000 7ff0f77fe640 1 -- 172.21.15.60:0/655264290 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2069761063 0 0) 0x7ff0f80b1650 con 0x7ff0f80aef70 2024-05-09T05:01:22.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.600+0000 7ff0f77fe640 1 -- 172.21.15.60:0/655264290 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0e8003520 con 0x7ff0f80aef70 2024-05-09T05:01:22.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0f77fe640 1 -- 172.21.15.60:0/655264290 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3745990472 0 0) 0x7ff0e8003520 con 0x7ff0f80aef70 2024-05-09T05:01:22.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0f77fe640 1 -- 172.21.15.60:0/655264290 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0f80b1110 msgr2=0x7ff0f80b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:22.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0f77fe640 1 --2- 172.21.15.60:0/655264290 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0f80b1110 0x7ff0f80b2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0f77fe640 1 -- 172.21.15.60:0/655264290 >> v2:172.21.15.114:3300/0 conn(0x7ff0f8096c90 msgr2=0x7ff0f80aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:22.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0f77fe640 1 --2- 172.21.15.60:0/655264290 >> v2:172.21.15.114:3300/0 conn(0x7ff0f8096c90 0x7ff0f80aea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0f77fe640 1 -- 172.21.15.60:0/655264290 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f80b1930 con 0x7ff0f80aef70 2024-05-09T05:01:22.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0f77fe640 1 -- 172.21.15.60:0/655264290 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7ff0e4001710 con 0x7ff0f80aef70 2024-05-09T05:01:22.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0f77fe640 1 -- 172.21.15.60:0/655264290 >> v1:172.21.15.60:6789/0 conn(0x7ff0f80aef70 legacy=0x7ff0f80b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:22.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0f77fe640 1 --2- 172.21.15.60:0/655264290 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0f80b1110 0x7ff0e8003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0f77fe640 1 --2- 172.21.15.60:0/655264290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0e8004020 0x7ff0e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0f77fe640 1 --2- 172.21.15.60:0/655264290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0e8004d40 0x7ff0e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0f77fe640 1 -- 172.21.15.60:0/655264290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f8160a90 con 0x7ff0e8004d40 2024-05-09T05:01:22.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0f77fe640 1 -- 172.21.15.60:0/655264290 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f80b1fb0 con 0x7ff0e8004020 2024-05-09T05:01:22.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0f77fe640 1 -- 172.21.15.60:0/655264290 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff0e800bba0 con 0x7ff0f80b1110 2024-05-09T05:01:22.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0fd619640 1 --2- 172.21.15.60:0/655264290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0e8004d40 0x7ff0e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0f7fff640 1 --2- 172.21.15.60:0/655264290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0e8004020 0x7ff0e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0fce18640 1 --2- 172.21.15.60:0/655264290 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0f80b1110 0x7ff0e8003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0fd619640 1 -- 172.21.15.60:0/655264290 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0f80b1110 msgr2=0x7ff0e8003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:22.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0fd619640 1 --2- 172.21.15.60:0/655264290 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0f80b1110 0x7ff0e8003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0fd619640 1 -- 172.21.15.60:0/655264290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0e8004020 msgr2=0x7ff0e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:22.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0fd619640 1 --2- 172.21.15.60:0/655264290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0e8004020 0x7ff0e8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0fd619640 1 -- 172.21.15.60:0/655264290 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0e8003520 con 0x7ff0e8004d40 2024-05-09T05:01:22.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0f7fff640 1 --2- 172.21.15.60:0/655264290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0e8004020 0x7ff0e8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:22.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.601+0000 7ff0fce18640 1 --2- 172.21.15.60:0/655264290 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0f80b1110 0x7ff0e8003a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:22.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.602+0000 7ff0fd619640 1 --2- 172.21.15.60:0/655264290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0e8004d40 0x7ff0e80051c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff0f000e170 tx=0x7ff0e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b50e2a80983ef1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:22.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.602+0000 7ff0f77fe640 1 -- 172.21.15.60:0/655264290 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0034020 con 0x7ff0e8004d40 2024-05-09T05:01:22.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.602+0000 7ff0f77fe640 1 -- 172.21.15.60:0/655264290 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f00305f0 con 0x7ff0e8004d40 2024-05-09T05:01:22.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.602+0000 7ff0f77fe640 1 -- 172.21.15.60:0/655264290 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0009c10 con 0x7ff0e8004d40 2024-05-09T05:01:22.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.603+0000 7ff0ff07c640 1 -- 172.21.15.60:0/655264290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0e8004d40 msgr2=0x7ff0e80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:22.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.603+0000 7ff0ff07c640 1 --2- 172.21.15.60:0/655264290 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0e8004d40 0x7ff0e80051c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff0f000e170 tx=0x7ff0e8005830 comp rx=0 tx=0).stop 2024-05-09T05:01:22.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.603+0000 7ff0fd619640 1 -- 172.21.15.60:0/655264290 reap_dead start 2024-05-09T05:01:22.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.603+0000 7ff0ff07c640 1 -- 172.21.15.60:0/655264290 shutdown_connections 2024-05-09T05:01:22.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.603+0000 7ff0ff07c640 1 -- 172.21.15.60:0/655264290 >> 172.21.15.60:0/655264290 conn(0x7ff0f80722b0 msgr2=0x7ff0f8072690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:22.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.603+0000 7ff0ff07c640 1 -- 172.21.15.60:0/655264290 shutdown_connections 2024-05-09T05:01:22.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.603+0000 7ff0ff07c640 1 -- 172.21.15.60:0/655264290 wait complete. 2024-05-09T05:01:22.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.604+0000 7ff0ff07c640 1 Processor -- start 2024-05-09T05:01:22.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.604+0000 7ff0ff07c640 1 -- start start 2024-05-09T05:01:22.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.605+0000 7ff0ff07c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0e8004d40 0x7ff0f806cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.605+0000 7ff0ff07c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f80a9d60 0x7ff0f815d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.605+0000 7ff0ff07c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0f815dab0 0x7ff0f8098ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.605+0000 7ff0ff07c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f8097070 con 0x7ff0f815dab0 2024-05-09T05:01:22.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.605+0000 7ff0ff07c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f815fc80 con 0x7ff0f80a9d60 2024-05-09T05:01:22.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.605+0000 7ff0ff07c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff0f8099f00 con 0x7ff0e8004d40 2024-05-09T05:01:22.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.605+0000 7ff0fd619640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0f815dab0 0x7ff0f8098ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.605+0000 7ff0fd619640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0f815dab0 0x7ff0f8098ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59650/0 (socket says 172.21.15.60:59650) 2024-05-09T05:01:22.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.605+0000 7ff0fd619640 1 -- 172.21.15.60:0/2869979826 learned_addr learned my addr 172.21.15.60:0/2869979826 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:22.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.605+0000 7ff0fce18640 1 --2- 172.21.15.60:0/2869979826 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0e8004d40 0x7ff0f806cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.605+0000 7ff0f7fff640 1 --2- 172.21.15.60:0/2869979826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f80a9d60 0x7ff0f815d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.606+0000 7ff0fd619640 1 -- 172.21.15.60:0/2869979826 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0e8004d40 msgr2=0x7ff0f806cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:22.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.606+0000 7ff0fd619640 1 --2- 172.21.15.60:0/2869979826 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0e8004d40 0x7ff0f806cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.606+0000 7ff0fd619640 1 -- 172.21.15.60:0/2869979826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f80a9d60 msgr2=0x7ff0f815d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:22.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.606+0000 7ff0fd619640 1 --2- 172.21.15.60:0/2869979826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f80a9d60 0x7ff0f815d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.606+0000 7ff0fd619640 1 -- 172.21.15.60:0/2869979826 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f806d4a0 con 0x7ff0f815dab0 2024-05-09T05:01:22.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.606+0000 7ff0fd619640 1 --2- 172.21.15.60:0/2869979826 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0f815dab0 0x7ff0f8098ed0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff0f000e170 tx=0x7ff0f0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:22.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.606+0000 7ff0f57fa640 1 -- 172.21.15.60:0/2869979826 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0040070 con 0x7ff0f815dab0 2024-05-09T05:01:22.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.606+0000 7ff0f57fa640 1 -- 172.21.15.60:0/2869979826 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f0003990 con 0x7ff0f815dab0 2024-05-09T05:01:22.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.606+0000 7ff0ff07c640 1 -- 172.21.15.60:0/2869979826 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0f8099c20 con 0x7ff0f815dab0 2024-05-09T05:01:22.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.606+0000 7ff0f57fa640 1 -- 172.21.15.60:0/2869979826 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0034040 con 0x7ff0f815dab0 2024-05-09T05:01:22.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.606+0000 7ff0ff07c640 1 -- 172.21.15.60:0/2869979826 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0f806c760 con 0x7ff0f815dab0 2024-05-09T05:01:22.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.607+0000 7ff0ff07c640 1 -- 172.21.15.60:0/2869979826 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0f8051040 con 0x7ff0f815dab0 2024-05-09T05:01:22.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.608+0000 7ff0f57fa640 1 -- 172.21.15.60:0/2869979826 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff0f0037020 con 0x7ff0f815dab0 2024-05-09T05:01:22.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.608+0000 7ff0f57fa640 1 --2- 172.21.15.60:0/2869979826 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff0dc0762d0 0x7ff0dc078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:22.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.608+0000 7ff0f57fa640 1 -- 172.21.15.60:0/2869979826 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6496+0+0 (secure 0 0 0) 0x7ff0f00b7c80 con 0x7ff0f815dab0 2024-05-09T05:01:22.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.609+0000 7ff0fce18640 1 --2- 172.21.15.60:0/2869979826 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff0dc0762d0 0x7ff0dc078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:22.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.609+0000 7ff0fce18640 1 --2- 172.21.15.60:0/2869979826 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff0dc0762d0 0x7ff0dc078790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec009b40 tx=0x7ff0ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:22.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.614+0000 7ff0f57fa640 1 -- 172.21.15.60:0/2869979826 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff0f0080c80 con 0x7ff0f815dab0 2024-05-09T05:01:22.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.818+0000 7ff0ff07c640 1 -- 172.21.15.60:0/2869979826 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff0f809b200 con 0x7ff0f815dab0 2024-05-09T05:01:22.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.819+0000 7ff0f57fa640 1 -- 172.21.15.60:0/2869979826 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v39) ==== 80+0+0 (secure 0 0 0) 0x7ff0f0084f80 con 0x7ff0f815dab0 2024-05-09T05:01:22.820 INFO:teuthology.orchestra.run.smithi060.stderr:listed 0 entries 2024-05-09T05:01:22.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.821+0000 7ff0ff07c640 1 -- 172.21.15.60:0/2869979826 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff0dc0762d0 msgr2=0x7ff0dc078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:22.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.821+0000 7ff0ff07c640 1 --2- 172.21.15.60:0/2869979826 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff0dc0762d0 0x7ff0dc078790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec009b40 tx=0x7ff0ec008040 comp rx=0 tx=0).stop 2024-05-09T05:01:22.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.821+0000 7ff0ff07c640 1 -- 172.21.15.60:0/2869979826 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0f815dab0 msgr2=0x7ff0f8098ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:22.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.821+0000 7ff0ff07c640 1 --2- 172.21.15.60:0/2869979826 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0f815dab0 0x7ff0f8098ed0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff0f000e170 tx=0x7ff0f0009750 comp rx=0 tx=0).stop 2024-05-09T05:01:22.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.821+0000 7ff0ff07c640 1 -- 172.21.15.60:0/2869979826 shutdown_connections 2024-05-09T05:01:22.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.821+0000 7ff0ff07c640 1 --2- 172.21.15.60:0/2869979826 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff0dc0762d0 0x7ff0dc078790 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.821+0000 7ff0ff07c640 1 --2- 172.21.15.60:0/2869979826 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0f815dab0 0x7ff0f8098ed0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.821+0000 7ff0ff07c640 1 --2- 172.21.15.60:0/2869979826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f80a9d60 0x7ff0f815d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.821+0000 7ff0ff07c640 1 --2- 172.21.15.60:0/2869979826 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff0e8004d40 0x7ff0f806cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:22.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.821+0000 7ff0ff07c640 1 -- 172.21.15.60:0/2869979826 >> 172.21.15.60:0/2869979826 conn(0x7ff0f80722b0 msgr2=0x7ff0f80aabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:22.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.821+0000 7ff0ff07c640 1 -- 172.21.15.60:0/2869979826 shutdown_connections 2024-05-09T05:01:22.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:22.821+0000 7ff0ff07c640 1 -- 172.21.15.60:0/2869979826 wait complete. 2024-05-09T05:01:22.832 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-09T05:01:22.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:22.833 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-09T05:01:22.889 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2024-05-09T05:01:22.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:22.890 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T05:01:22.913 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T05:01:22.913 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T05:01:22.913 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:01:22.930 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056322,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497656k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248824k,nr_inodes=812206,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497656k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T05:01:22.930 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T05:01:22.930 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:01:22.931 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:01:22.970 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057102,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498908k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19678 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498908k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T05:01:22.970 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T05:01:22.970 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T05:01:22.970 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:01:22.986 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056322,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497656k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248824k,nr_inodes=812206,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497656k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T05:01:22.986 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:23.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.114+0000 7f652fa45640 1 Processor -- start 2024-05-09T05:01:23.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.114+0000 7f652fa45640 1 -- start start 2024-05-09T05:01:23.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.115+0000 7f652fa45640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f652815f3c0 0x7f652809ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:23.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.115+0000 7f652fa45640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f652815f7a0 con 0x7f65280b4d30 2024-05-09T05:01:23.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.115+0000 7f652fa45640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f652809f480 con 0x7f6528073fb0 2024-05-09T05:01:23.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.115+0000 7f652fa45640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f6528071cb0 con 0x7f652815f3c0 2024-05-09T05:01:23.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.115+0000 7f652dfe2640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f65280b4d30 0x7f6528072140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42114/0 (socket says 172.21.15.60:42114) 2024-05-09T05:01:23.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.115+0000 7f652dfe2640 1 -- 172.21.15.60:0/1607291808 learned_addr learned my addr 172.21.15.60:0/1607291808 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:23.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.115+0000 7f652d7e1640 1 --2- 172.21.15.60:0/1607291808 >> v2:172.21.15.114:3300/0 conn(0x7f652815f3c0 0x7f652809ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:23.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.115+0000 7f651ffff640 1 -- 172.21.15.60:0/1607291808 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3113316613 0 0) 0x7f652815f7a0 con 0x7f65280b4d30 2024-05-09T05:01:23.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.115+0000 7f651ffff640 1 -- 172.21.15.60:0/1607291808 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6514003520 con 0x7f65280b4d30 2024-05-09T05:01:23.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.115+0000 7f652d7e1640 1 -- 172.21.15.60:0/1607291808 >> v1:172.21.15.114:6789/0 conn(0x7f6528073fb0 legacy=0x7f65280b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:23.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.115+0000 7f652d7e1640 1 -- 172.21.15.60:0/1607291808 >> v1:172.21.15.60:6789/0 conn(0x7f65280b4d30 legacy=0x7f6528072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:23.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.115+0000 7f652d7e1640 1 -- 172.21.15.60:0/1607291808 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f652809f760 con 0x7f652815f3c0 2024-05-09T05:01:23.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.116+0000 7f652d7e1640 1 --2- 172.21.15.60:0/1607291808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f652815f3c0 0x7f652809ebe0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f65240097a0 tx=0x7f652402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a8f81299c94310a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:23.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.116+0000 7f651ffff640 1 -- 172.21.15.60:0/1607291808 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6524037070 con 0x7f652815f3c0 2024-05-09T05:01:23.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.116+0000 7f651ffff640 1 -- 172.21.15.60:0/1607291808 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6524032440 con 0x7f652815f3c0 2024-05-09T05:01:23.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.116+0000 7f651ffff640 1 -- 172.21.15.60:0/1607291808 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6524032810 con 0x7f652815f3c0 2024-05-09T05:01:23.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.117+0000 7f652fa45640 1 -- 172.21.15.60:0/1607291808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f652815f3c0 msgr2=0x7f652809ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:23.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.117+0000 7f652fa45640 1 --2- 172.21.15.60:0/1607291808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f652815f3c0 0x7f652809ebe0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f65240097a0 tx=0x7f652402a1a0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.117+0000 7f652fa45640 1 -- 172.21.15.60:0/1607291808 shutdown_connections 2024-05-09T05:01:23.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.117+0000 7f652fa45640 1 --2- 172.21.15.60:0/1607291808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f652815f3c0 0x7f652809ebe0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.117+0000 7f652fa45640 1 -- 172.21.15.60:0/1607291808 >> 172.21.15.60:0/1607291808 conn(0x7f65280d1470 msgr2=0x7f65280d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:23.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.117+0000 7f652fa45640 1 -- 172.21.15.60:0/1607291808 shutdown_connections 2024-05-09T05:01:23.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.117+0000 7f652fa45640 1 -- 172.21.15.60:0/1607291808 wait complete. 2024-05-09T05:01:23.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.118+0000 7f652fa45640 1 Processor -- start 2024-05-09T05:01:23.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.118+0000 7f652fa45640 1 -- start start 2024-05-09T05:01:23.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.119+0000 7f652fa45640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6528073fb0 0x7f65280b2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:23.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.119+0000 7f652fa45640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65280b4d30 0x7f65280d4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:23.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.119+0000 7f652fa45640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f652815f3c0 0x7f65280d4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:23.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.119+0000 7f652fa45640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6528072850 con 0x7f65280b4d30 2024-05-09T05:01:23.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.119+0000 7f652fa45640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f652809f120 con 0x7f652815f3c0 2024-05-09T05:01:23.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.119+0000 7f652fa45640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f65280729d0 con 0x7f6528073fb0 2024-05-09T05:01:23.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.119+0000 7f652cfe0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65280b4d30 0x7f65280d4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:23.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.119+0000 7f652cfe0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65280b4d30 0x7f65280d4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59660/0 (socket says 172.21.15.60:59660) 2024-05-09T05:01:23.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.119+0000 7f652cfe0640 1 -- 172.21.15.60:0/1909594577 learned_addr learned my addr 172.21.15.60:0/1909594577 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:23.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.119+0000 7f652dfe2640 1 --2- 172.21.15.60:0/1909594577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f652815f3c0 0x7f65280d4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:23.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.119+0000 7f652d7e1640 1 --2- 172.21.15.60:0/1909594577 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6528073fb0 0x7f65280b2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:23.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.119+0000 7f652cfe0640 1 -- 172.21.15.60:0/1909594577 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6528073fb0 msgr2=0x7f65280b2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:23.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.119+0000 7f652cfe0640 1 --2- 172.21.15.60:0/1909594577 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6528073fb0 0x7f65280b2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.119+0000 7f652cfe0640 1 -- 172.21.15.60:0/1909594577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f652815f3c0 msgr2=0x7f65280d4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:23.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.119+0000 7f652cfe0640 1 --2- 172.21.15.60:0/1909594577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f652815f3c0 0x7f65280d4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.119+0000 7f652cfe0640 1 -- 172.21.15.60:0/1909594577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65280af5d0 con 0x7f65280b4d30 2024-05-09T05:01:23.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.119+0000 7f652cfe0640 1 --2- 172.21.15.60:0/1909594577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65280b4d30 0x7f65280d4000 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f651800c500 tx=0x7f651800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:23.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.120+0000 7f651dffb640 1 -- 172.21.15.60:0/1909594577 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65180052d0 con 0x7f65280b4d30 2024-05-09T05:01:23.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.120+0000 7f651dffb640 1 -- 172.21.15.60:0/1909594577 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6518005470 con 0x7f65280b4d30 2024-05-09T05:01:23.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.120+0000 7f652fa45640 1 -- 172.21.15.60:0/1909594577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f652814eee0 con 0x7f65280b4d30 2024-05-09T05:01:23.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.120+0000 7f651dffb640 1 -- 172.21.15.60:0/1909594577 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6518012870 con 0x7f65280b4d30 2024-05-09T05:01:23.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.120+0000 7f652fa45640 1 -- 172.21.15.60:0/1909594577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f652814f4a0 con 0x7f65280b4d30 2024-05-09T05:01:23.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.121+0000 7f652fa45640 1 -- 172.21.15.60:0/1909594577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6528051040 con 0x7f65280b4d30 2024-05-09T05:01:23.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.121+0000 7f651dffb640 1 -- 172.21.15.60:0/1909594577 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6518012a10 con 0x7f65280b4d30 2024-05-09T05:01:23.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.122+0000 7f651dffb640 1 --2- 172.21.15.60:0/1909594577 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f650c0762d0 0x7f650c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:23.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.122+0000 7f651dffb640 1 -- 172.21.15.60:0/1909594577 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6496+0+0 (secure 0 0 0) 0x7f65180973f0 con 0x7f65280b4d30 2024-05-09T05:01:23.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.122+0000 7f652d7e1640 1 --2- 172.21.15.60:0/1909594577 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f650c0762d0 0x7f650c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:23.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.122+0000 7f652d7e1640 1 --2- 172.21.15.60:0/1909594577 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f650c0762d0 0x7f650c078790 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6524034000 tx=0x7f652402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:23.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.126+0000 7f651dffb640 1 -- 172.21.15.60:0/1909594577 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6518060400 con 0x7f65280b4d30 2024-05-09T05:01:23.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.374+0000 7f652fa45640 1 -- 172.21.15.60:0/1909594577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65280701a0 con 0x7f65280b4d30 2024-05-09T05:01:23.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.374+0000 7f651dffb640 1 -- 172.21.15.60:0/1909594577 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5183 (secure 0 0 0) 0x7f6518064700 con 0x7f65280b4d30 2024-05-09T05:01:23.375 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:23.375 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4547,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2372252858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2372252858},{"type":"v1","addr":"172.21.15.114:6836","nonce":2372252858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1761282610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1761282610},{"type":"v1","addr":"172.21.15.114:6837","nonce":1761282610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4765,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/300408012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":300408012},{"type":"v1","addr":"172.21.15.60:6835","nonce":300408012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:04.698975+0000","modified":"2024-05-09T05:01:18.144520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4893},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4893":{"gid":4893,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/399160914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":399160914},{"type":"v1","addr":"172.21.15.60:6834","nonce":399160914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4893,"qdb_cluster":[4893]},"id":2}]} 2024-05-09T05:01:23.376 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 18 2024-05-09T05:01:23.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.377+0000 7f652fa45640 1 -- 172.21.15.60:0/1909594577 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f650c0762d0 msgr2=0x7f650c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:23.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.377+0000 7f652fa45640 1 --2- 172.21.15.60:0/1909594577 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f650c0762d0 0x7f650c078790 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6524034000 tx=0x7f652402e000 comp rx=0 tx=0).stop 2024-05-09T05:01:23.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.377+0000 7f652fa45640 1 -- 172.21.15.60:0/1909594577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65280b4d30 msgr2=0x7f65280d4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:23.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.377+0000 7f652fa45640 1 --2- 172.21.15.60:0/1909594577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65280b4d30 0x7f65280d4000 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f651800c500 tx=0x7f651800c9c0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.377+0000 7f652fa45640 1 -- 172.21.15.60:0/1909594577 shutdown_connections 2024-05-09T05:01:23.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.377+0000 7f652fa45640 1 --2- 172.21.15.60:0/1909594577 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f650c0762d0 0x7f650c078790 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.377+0000 7f652fa45640 1 --2- 172.21.15.60:0/1909594577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f652815f3c0 0x7f65280d4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.377+0000 7f652fa45640 1 --2- 172.21.15.60:0/1909594577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65280b4d30 0x7f65280d4000 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.377+0000 7f652fa45640 1 --2- 172.21.15.60:0/1909594577 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6528073fb0 0x7f65280b2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.377+0000 7f652fa45640 1 -- 172.21.15.60:0/1909594577 >> 172.21.15.60:0/1909594577 conn(0x7f65280d1470 msgr2=0x7f6528099f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:23.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.377+0000 7f652fa45640 1 -- 172.21.15.60:0/1909594577 shutdown_connections 2024-05-09T05:01:23.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.377+0000 7f652fa45640 1 -- 172.21.15.60:0/1909594577 wait complete. 2024-05-09T05:01:23.389 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:23.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.492+0000 7fe3f0a3e640 1 Processor -- start 2024-05-09T05:01:23.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.492+0000 7fe3f0a3e640 1 -- start start 2024-05-09T05:01:23.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.493+0000 7fe3f0a3e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe3ec0ac430 0x7fe3ec073330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:23.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.493+0000 7fe3f0a3e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3ec071b80 con 0x7fe3ec070860 2024-05-09T05:01:23.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.493+0000 7fe3f0a3e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3ec071e60 con 0x7fe3ec15c3a0 2024-05-09T05:01:23.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.493+0000 7fe3f0a3e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fe3ec072140 con 0x7fe3ec0ac430 2024-05-09T05:01:23.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.493+0000 7fe3ead9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe3ec0ac430 0x7fe3ec073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:23.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.494+0000 7fe3e9d9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe3ec070860 0x7fe3ec072c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42122/0 (socket says 172.21.15.60:42122) 2024-05-09T05:01:23.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.494+0000 7fe3e9d9b640 1 -- 172.21.15.60:0/2127620629 learned_addr learned my addr 172.21.15.60:0/2127620629 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:23.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.494+0000 7fe3e959a640 1 -- 172.21.15.60:0/2127620629 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3818399532 0 0) 0x7fe3ec071b80 con 0x7fe3ec070860 2024-05-09T05:01:23.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.494+0000 7fe3e959a640 1 -- 172.21.15.60:0/2127620629 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3d4003520 con 0x7fe3ec070860 2024-05-09T05:01:23.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.494+0000 7fe3ead9d640 1 -- 172.21.15.60:0/2127620629 >> v1:172.21.15.114:6789/0 conn(0x7fe3ec15c3a0 legacy=0x7fe3ec070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:23.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.494+0000 7fe3ead9d640 1 -- 172.21.15.60:0/2127620629 >> v1:172.21.15.60:6789/0 conn(0x7fe3ec070860 legacy=0x7fe3ec072c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:23.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.494+0000 7fe3ead9d640 1 -- 172.21.15.60:0/2127620629 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3ec072420 con 0x7fe3ec0ac430 2024-05-09T05:01:23.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.495+0000 7fe3ead9d640 1 --2- 172.21.15.60:0/2127620629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3ec0ac430 0x7fe3ec073330 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe3e400bfa0 tx=0x7fe3e402c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f8734666ce64d2e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:23.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.495+0000 7fe3e959a640 1 -- 172.21.15.60:0/2127620629 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4009540 con 0x7fe3ec0ac430 2024-05-09T05:01:23.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.495+0000 7fe3e959a640 1 -- 172.21.15.60:0/2127620629 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3e4037070 con 0x7fe3ec0ac430 2024-05-09T05:01:23.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.495+0000 7fe3e959a640 1 -- 172.21.15.60:0/2127620629 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4009900 con 0x7fe3ec0ac430 2024-05-09T05:01:23.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.495+0000 7fe3f0a3e640 1 -- 172.21.15.60:0/2127620629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3ec0ac430 msgr2=0x7fe3ec073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:23.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.495+0000 7fe3f0a3e640 1 --2- 172.21.15.60:0/2127620629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3ec0ac430 0x7fe3ec073330 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe3e400bfa0 tx=0x7fe3e402c900 comp rx=0 tx=0).stop 2024-05-09T05:01:23.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.495+0000 7fe3f0a3e640 1 -- 172.21.15.60:0/2127620629 shutdown_connections 2024-05-09T05:01:23.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.495+0000 7fe3f0a3e640 1 --2- 172.21.15.60:0/2127620629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3ec0ac430 0x7fe3ec073330 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.495+0000 7fe3f0a3e640 1 -- 172.21.15.60:0/2127620629 >> 172.21.15.60:0/2127620629 conn(0x7fe3ec0972f0 msgr2=0x7fe3ec0976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:23.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.495+0000 7fe3f0a3e640 1 -- 172.21.15.60:0/2127620629 shutdown_connections 2024-05-09T05:01:23.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.495+0000 7fe3f0a3e640 1 -- 172.21.15.60:0/2127620629 wait complete. 2024-05-09T05:01:23.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.497+0000 7fe3f0a3e640 1 Processor -- start 2024-05-09T05:01:23.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.497+0000 7fe3f0a3e640 1 -- start start 2024-05-09T05:01:23.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3f0a3e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3ec070860 0x7fe3ec08f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:23.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3f0a3e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3ec0ac430 0x7fe3ec1f5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:23.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3f0a3e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3ec15c3a0 0x7fe3ec0abd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:23.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3f0a3e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec071810 con 0x7fe3ec0ac430 2024-05-09T05:01:23.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3f0a3e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec15c780 con 0x7fe3ec070860 2024-05-09T05:01:23.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3f0a3e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe3ec097de0 con 0x7fe3ec15c3a0 2024-05-09T05:01:23.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3e9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3ec0ac430 0x7fe3ec1f5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:23.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3e9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3ec0ac430 0x7fe3ec1f5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59676/0 (socket says 172.21.15.60:59676) 2024-05-09T05:01:23.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3e9d9b640 1 -- 172.21.15.60:0/4108205364 learned_addr learned my addr 172.21.15.60:0/4108205364 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:23.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3ea59c640 1 --2- 172.21.15.60:0/4108205364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3ec070860 0x7fe3ec08f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:23.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3ead9d640 1 --2- 172.21.15.60:0/4108205364 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3ec15c3a0 0x7fe3ec0abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:23.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3e9d9b640 1 -- 172.21.15.60:0/4108205364 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3ec15c3a0 msgr2=0x7fe3ec0abd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:23.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3e9d9b640 1 --2- 172.21.15.60:0/4108205364 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3ec15c3a0 0x7fe3ec0abd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3e9d9b640 1 -- 172.21.15.60:0/4108205364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3ec070860 msgr2=0x7fe3ec08f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:23.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3e9d9b640 1 --2- 172.21.15.60:0/4108205364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3ec070860 0x7fe3ec08f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3e9d9b640 1 -- 172.21.15.60:0/4108205364 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3ec1f4a50 con 0x7fe3ec0ac430 2024-05-09T05:01:23.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3e9d9b640 1 --2- 172.21.15.60:0/4108205364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3ec0ac430 0x7fe3ec1f5650 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000c430 tx=0x7fe3e000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:23.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3daffd640 1 -- 172.21.15.60:0/4108205364 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0017070 con 0x7fe3ec0ac430 2024-05-09T05:01:23.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3daffd640 1 -- 172.21.15.60:0/4108205364 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3e0005d00 con 0x7fe3ec0ac430 2024-05-09T05:01:23.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3daffd640 1 -- 172.21.15.60:0/4108205364 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0007310 con 0x7fe3ec0ac430 2024-05-09T05:01:23.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.498+0000 7fe3f0a3e640 1 -- 172.21.15.60:0/4108205364 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3ec08fd80 con 0x7fe3ec0ac430 2024-05-09T05:01:23.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.499+0000 7fe3f0a3e640 1 -- 172.21.15.60:0/4108205364 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3ec090340 con 0x7fe3ec0ac430 2024-05-09T05:01:23.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.500+0000 7fe3f0a3e640 1 -- 172.21.15.60:0/4108205364 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3ec051040 con 0x7fe3ec0ac430 2024-05-09T05:01:23.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.500+0000 7fe3daffd640 1 -- 172.21.15.60:0/4108205364 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe3e001a070 con 0x7fe3ec0ac430 2024-05-09T05:01:23.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.501+0000 7fe3daffd640 1 --2- 172.21.15.60:0/4108205364 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe3cc0762d0 0x7fe3cc078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:23.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.501+0000 7fe3daffd640 1 -- 172.21.15.60:0/4108205364 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6496+0+0 (secure 0 0 0) 0x7fe3e0096cb0 con 0x7fe3ec0ac430 2024-05-09T05:01:23.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.501+0000 7fe3ea59c640 1 --2- 172.21.15.60:0/4108205364 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe3cc0762d0 0x7fe3cc078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:23.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.501+0000 7fe3ea59c640 1 --2- 172.21.15.60:0/4108205364 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe3cc0762d0 0x7fe3cc078790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc000f60 tx=0x7fe3dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:23.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.505+0000 7fe3daffd640 1 -- 172.21.15.60:0/4108205364 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe3e005fcb0 con 0x7fe3ec0ac430 2024-05-09T05:01:23.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.733+0000 7fe3f0a3e640 1 -- 172.21.15.60:0/4108205364 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3ec15b140 con 0x7fe3ec0ac430 2024-05-09T05:01:23.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.734+0000 7fe3daffd640 1 -- 172.21.15.60:0/4108205364 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5183 (secure 0 0 0) 0x7fe3e0063fb0 con 0x7fe3ec0ac430 2024-05-09T05:01:23.734 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:23.735 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4547,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2372252858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2372252858},{"type":"v1","addr":"172.21.15.114:6836","nonce":2372252858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1761282610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1761282610},{"type":"v1","addr":"172.21.15.114:6837","nonce":1761282610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4765,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/300408012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":300408012},{"type":"v1","addr":"172.21.15.60:6835","nonce":300408012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:04.698975+0000","modified":"2024-05-09T05:01:18.144520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4893},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4893":{"gid":4893,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/399160914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":399160914},{"type":"v1","addr":"172.21.15.60:6834","nonce":399160914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4893,"qdb_cluster":[4893]},"id":2}]} 2024-05-09T05:01:23.735 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 18 2024-05-09T05:01:23.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.735+0000 7fe3f0a3e640 1 -- 172.21.15.60:0/4108205364 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe3cc0762d0 msgr2=0x7fe3cc078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:23.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.735+0000 7fe3f0a3e640 1 --2- 172.21.15.60:0/4108205364 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe3cc0762d0 0x7fe3cc078790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc000f60 tx=0x7fe3dc009040 comp rx=0 tx=0).stop 2024-05-09T05:01:23.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.735+0000 7fe3f0a3e640 1 -- 172.21.15.60:0/4108205364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3ec0ac430 msgr2=0x7fe3ec1f5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:23.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.735+0000 7fe3f0a3e640 1 --2- 172.21.15.60:0/4108205364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3ec0ac430 0x7fe3ec1f5650 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000c430 tx=0x7fe3e000c8f0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.736+0000 7fe3f0a3e640 1 -- 172.21.15.60:0/4108205364 shutdown_connections 2024-05-09T05:01:23.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.736+0000 7fe3f0a3e640 1 --2- 172.21.15.60:0/4108205364 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe3cc0762d0 0x7fe3cc078790 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.736+0000 7fe3f0a3e640 1 --2- 172.21.15.60:0/4108205364 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe3ec15c3a0 0x7fe3ec0abd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.736+0000 7fe3f0a3e640 1 --2- 172.21.15.60:0/4108205364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3ec0ac430 0x7fe3ec1f5650 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.736+0000 7fe3f0a3e640 1 --2- 172.21.15.60:0/4108205364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3ec070860 0x7fe3ec08f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.736+0000 7fe3f0a3e640 1 -- 172.21.15.60:0/4108205364 >> 172.21.15.60:0/4108205364 conn(0x7fe3ec0972f0 msgr2=0x7fe3ec099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:23.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.736+0000 7fe3f0a3e640 1 -- 172.21.15.60:0/4108205364 shutdown_connections 2024-05-09T05:01:23.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.736+0000 7fe3f0a3e640 1 -- 172.21.15.60:0/4108205364 wait complete. 2024-05-09T05:01:23.749 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:01:23.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.856+0000 7fd67b283640 1 Processor -- start 2024-05-09T05:01:23.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.856+0000 7fd67b283640 1 -- start start 2024-05-09T05:01:23.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.857+0000 7fd67b283640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fd6740d1050 0x7fd6740ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:23.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.857+0000 7fd67b283640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6740d1430 0x7fd6740d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:23.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.857+0000 7fd67b283640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6740af370 con 0x7fd674096eb0 2024-05-09T05:01:23.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.857+0000 7fd67b283640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fd674096a00 con 0x7fd6740d1050 2024-05-09T05:01:23.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.857+0000 7fd67b283640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd6740d2dd0 con 0x7fd6740d1430 2024-05-09T05:01:23.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.857+0000 7fd67881e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd674096eb0 0x7fd67409b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42132/0 (socket says 172.21.15.60:42132) 2024-05-09T05:01:23.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.857+0000 7fd67881e640 1 -- 172.21.15.60:0/2195970434 learned_addr learned my addr 172.21.15.60:0/2195970434 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:23.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.857+0000 7fd67901f640 1 --2- 172.21.15.60:0/2195970434 >> v2:172.21.15.114:3300/0 conn(0x7fd6740d1050 0x7fd6740ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:23.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.857+0000 7fd679820640 1 --2- 172.21.15.60:0/2195970434 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6740d1430 0x7fd6740d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:23.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.857+0000 7fd667fff640 1 -- 172.21.15.60:0/2195970434 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4190624370 0 0) 0x7fd6740af370 con 0x7fd674096eb0 2024-05-09T05:01:23.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.857+0000 7fd667fff640 1 -- 172.21.15.60:0/2195970434 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd660003520 con 0x7fd674096eb0 2024-05-09T05:01:23.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.858+0000 7fd67901f640 1 -- 172.21.15.60:0/2195970434 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6740d1430 msgr2=0x7fd6740d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:23.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.858+0000 7fd67901f640 1 --2- 172.21.15.60:0/2195970434 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6740d1430 0x7fd6740d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.858+0000 7fd67901f640 1 -- 172.21.15.60:0/2195970434 >> v1:172.21.15.60:6789/0 conn(0x7fd674096eb0 legacy=0x7fd67409b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:23.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.858+0000 7fd67901f640 1 -- 172.21.15.60:0/2195970434 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6740d30b0 con 0x7fd6740d1050 2024-05-09T05:01:23.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.858+0000 7fd679820640 1 --2- 172.21.15.60:0/2195970434 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6740d1430 0x7fd6740d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:01:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.858+0000 7fd67901f640 1 --2- 172.21.15.60:0/2195970434 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6740d1050 0x7fd6740ae9f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd6700097a0 tx=0x7fd67002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9181151e9d21171c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.859+0000 7fd667fff640 1 -- 172.21.15.60:0/2195970434 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd670037070 con 0x7fd6740d1050 2024-05-09T05:01:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.859+0000 7fd667fff640 1 -- 172.21.15.60:0/2195970434 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd670032440 con 0x7fd6740d1050 2024-05-09T05:01:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.859+0000 7fd667fff640 1 -- 172.21.15.60:0/2195970434 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6700327f0 con 0x7fd6740d1050 2024-05-09T05:01:23.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.859+0000 7fd67b283640 1 -- 172.21.15.60:0/2195970434 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6740d1050 msgr2=0x7fd6740ae9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:23.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.859+0000 7fd67b283640 1 --2- 172.21.15.60:0/2195970434 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6740d1050 0x7fd6740ae9f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd6700097a0 tx=0x7fd67002a180 comp rx=0 tx=0).stop 2024-05-09T05:01:23.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.859+0000 7fd67b283640 1 -- 172.21.15.60:0/2195970434 shutdown_connections 2024-05-09T05:01:23.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.859+0000 7fd67b283640 1 --2- 172.21.15.60:0/2195970434 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd6740d1430 0x7fd6740d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.859+0000 7fd67b283640 1 --2- 172.21.15.60:0/2195970434 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6740d1050 0x7fd6740ae9f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.859+0000 7fd67b283640 1 -- 172.21.15.60:0/2195970434 >> 172.21.15.60:0/2195970434 conn(0x7fd6740993f0 msgr2=0x7fd674099800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:23.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.859+0000 7fd67b283640 1 -- 172.21.15.60:0/2195970434 shutdown_connections 2024-05-09T05:01:23.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.859+0000 7fd67b283640 1 -- 172.21.15.60:0/2195970434 wait complete. 2024-05-09T05:01:23.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.861+0000 7fd67b283640 1 Processor -- start 2024-05-09T05:01:23.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.861+0000 7fd67b283640 1 -- start start 2024-05-09T05:01:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.861+0000 7fd67b283640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd674096eb0 0x7fd67406f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.861+0000 7fd67b283640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6740d1050 0x7fd67406f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.861+0000 7fd67b283640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6740d1430 0x7fd6741da090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.861+0000 7fd67b283640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd67409c1e0 con 0x7fd6740d1050 2024-05-09T05:01:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.861+0000 7fd67b283640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd67409c060 con 0x7fd6740d1430 2024-05-09T05:01:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.861+0000 7fd67b283640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd6740d2790 con 0x7fd674096eb0 2024-05-09T05:01:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.861+0000 7fd67881e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6740d1050 0x7fd67406f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:23.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.861+0000 7fd67881e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6740d1050 0x7fd67406f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59690/0 (socket says 172.21.15.60:59690) 2024-05-09T05:01:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.862+0000 7fd67881e640 1 -- 172.21.15.60:0/2151088003 learned_addr learned my addr 172.21.15.60:0/2151088003 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.862+0000 7fd67901f640 1 --2- 172.21.15.60:0/2151088003 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd674096eb0 0x7fd67406f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.862+0000 7fd679820640 1 --2- 172.21.15.60:0/2151088003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6740d1430 0x7fd6741da090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.862+0000 7fd67881e640 1 -- 172.21.15.60:0/2151088003 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd674096eb0 msgr2=0x7fd67406f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.862+0000 7fd67881e640 1 --2- 172.21.15.60:0/2151088003 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd674096eb0 0x7fd67406f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.862+0000 7fd67881e640 1 -- 172.21.15.60:0/2151088003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6740d1430 msgr2=0x7fd6741da090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.862+0000 7fd67881e640 1 --2- 172.21.15.60:0/2151088003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6740d1430 0x7fd6741da090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.862+0000 7fd67881e640 1 -- 172.21.15.60:0/2151088003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6741d4c20 con 0x7fd6740d1050 2024-05-09T05:01:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.862+0000 7fd67881e640 1 --2- 172.21.15.60:0/2151088003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6740d1050 0x7fd67406f5b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd66800c440 tx=0x7fd66800c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.862+0000 7fd665ffb640 1 -- 172.21.15.60:0/2151088003 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd668017070 con 0x7fd6740d1050 2024-05-09T05:01:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.862+0000 7fd665ffb640 1 -- 172.21.15.60:0/2151088003 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd668005ce0 con 0x7fd6740d1050 2024-05-09T05:01:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.862+0000 7fd67b283640 1 -- 172.21.15.60:0/2151088003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6741f4d00 con 0x7fd6740d1050 2024-05-09T05:01:23.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.862+0000 7fd665ffb640 1 -- 172.21.15.60:0/2151088003 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6680072d0 con 0x7fd6740d1050 2024-05-09T05:01:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.862+0000 7fd67b283640 1 -- 172.21.15.60:0/2151088003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6741d5170 con 0x7fd6740d1050 2024-05-09T05:01:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.863+0000 7fd67b283640 1 -- 172.21.15.60:0/2151088003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd674075540 con 0x7fd6740d1050 2024-05-09T05:01:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.864+0000 7fd665ffb640 1 -- 172.21.15.60:0/2151088003 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd66801a070 con 0x7fd6740d1050 2024-05-09T05:01:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.864+0000 7fd665ffb640 1 --2- 172.21.15.60:0/2151088003 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd6580762d0 0x7fd658078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.864+0000 7fd665ffb640 1 -- 172.21.15.60:0/2151088003 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6496+0+0 (secure 0 0 0) 0x7fd668096b90 con 0x7fd6740d1050 2024-05-09T05:01:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.865+0000 7fd67901f640 1 --2- 172.21.15.60:0/2151088003 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd6580762d0 0x7fd658078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:23.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.865+0000 7fd67901f640 1 --2- 172.21.15.60:0/2151088003 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd6580762d0 0x7fd658078790 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd670034000 tx=0x7fd67002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:23.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:23.869+0000 7fd665ffb640 1 -- 172.21.15.60:0/2151088003 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd6741f4d00 con 0x7fd6740d1050 2024-05-09T05:01:24.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.061+0000 7fd67b283640 1 -- 172.21.15.60:0/2151088003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd6740af070 con 0x7fd6740d1050 2024-05-09T05:01:24.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.062+0000 7fd665ffb640 1 -- 172.21.15.60:0/2151088003 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+18730 (secure 0 0 0) 0x7fd66805fb10 con 0x7fd6740d1050 2024-05-09T05:01:24.063 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:24.063 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":39,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:01:19.767056+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:01:02.773384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-05-09T05:01:03.658182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-05-09T05:01:16.847013+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:01:24.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.064+0000 7fd67b283640 1 -- 172.21.15.60:0/2151088003 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd6580762d0 msgr2=0x7fd658078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:24.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.064+0000 7fd67b283640 1 --2- 172.21.15.60:0/2151088003 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd6580762d0 0x7fd658078790 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd670034000 tx=0x7fd67002e000 comp rx=0 tx=0).stop 2024-05-09T05:01:24.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.064+0000 7fd67b283640 1 -- 172.21.15.60:0/2151088003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6740d1050 msgr2=0x7fd67406f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:24.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.064+0000 7fd67b283640 1 --2- 172.21.15.60:0/2151088003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6740d1050 0x7fd67406f5b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd66800c440 tx=0x7fd66800c840 comp rx=0 tx=0).stop 2024-05-09T05:01:24.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.064+0000 7fd67b283640 1 -- 172.21.15.60:0/2151088003 shutdown_connections 2024-05-09T05:01:24.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.064+0000 7fd67b283640 1 --2- 172.21.15.60:0/2151088003 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd6580762d0 0x7fd658078790 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.064+0000 7fd67b283640 1 --2- 172.21.15.60:0/2151088003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6740d1430 0x7fd6741da090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.064+0000 7fd67b283640 1 --2- 172.21.15.60:0/2151088003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6740d1050 0x7fd67406f5b0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.064+0000 7fd67b283640 1 --2- 172.21.15.60:0/2151088003 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd674096eb0 0x7fd67406f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.064+0000 7fd67b283640 1 -- 172.21.15.60:0/2151088003 >> 172.21.15.60:0/2151088003 conn(0x7fd6740993f0 msgr2=0x7fd6740b02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:24.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.064+0000 7fd67b283640 1 -- 172.21.15.60:0/2151088003 shutdown_connections 2024-05-09T05:01:24.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.064+0000 7fd67b283640 1 -- 172.21.15.60:0/2151088003 wait complete. 2024-05-09T05:01:24.076 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T05:01:24.076 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:24.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.180+0000 7f9de5746640 1 Processor -- start 2024-05-09T05:01:24.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.180+0000 7f9de5746640 1 -- start start 2024-05-09T05:01:24.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.181+0000 7f9de5746640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9de0098af0 0x7f9de009b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.181+0000 7f9de5746640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9de0098ed0 0x7f9de00b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.181+0000 7f9de5746640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de00d05e0 con 0x7f9de00d0070 2024-05-09T05:01:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.181+0000 7f9de5746640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f9de00ae9b0 con 0x7f9de0098ed0 2024-05-09T05:01:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.181+0000 7f9de5746640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9de00aec90 con 0x7f9de0098af0 2024-05-09T05:01:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.181+0000 7f9ddf7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9de00d0070 0x7f9de0163210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42140/0 (socket says 172.21.15.60:42140) 2024-05-09T05:01:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.181+0000 7f9ddf7fe640 1 -- 172.21.15.60:0/29709123 learned_addr learned my addr 172.21.15.60:0/29709123 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.181+0000 7f9dd7fff640 1 --2- 172.21.15.60:0/29709123 >> v2:172.21.15.114:3300/0 conn(0x7f9de0098ed0 0x7f9de00b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.181+0000 7f9dde7fc640 1 -- 172.21.15.60:0/29709123 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1387666907 0 0) 0x7f9de00d05e0 con 0x7f9de00d0070 2024-05-09T05:01:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.181+0000 7f9ddeffd640 1 --2- 172.21.15.60:0/29709123 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9de0098af0 0x7f9de009b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.181+0000 7f9dde7fc640 1 -- 172.21.15.60:0/29709123 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dcc003520 con 0x7f9de00d0070 2024-05-09T05:01:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.181+0000 7f9dde7fc640 1 -- 172.21.15.60:0/29709123 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2752720801 0 0) 0x7f9dcc003520 con 0x7f9de00d0070 2024-05-09T05:01:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.181+0000 7f9dde7fc640 1 -- 172.21.15.60:0/29709123 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9de0098af0 msgr2=0x7f9de009b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:24.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.181+0000 7f9dde7fc640 1 --2- 172.21.15.60:0/29709123 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9de0098af0 0x7f9de009b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.181+0000 7f9dde7fc640 1 -- 172.21.15.60:0/29709123 >> v2:172.21.15.114:3300/0 conn(0x7f9de0098ed0 msgr2=0x7f9de00b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.181+0000 7f9dde7fc640 1 --2- 172.21.15.60:0/29709123 >> v2:172.21.15.114:3300/0 conn(0x7f9de0098ed0 0x7f9de00b2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.181+0000 7f9dde7fc640 1 -- 172.21.15.60:0/29709123 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de00aef70 con 0x7f9de00d0070 2024-05-09T05:01:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.181+0000 7f9dde7fc640 1 -- 172.21.15.60:0/29709123 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f9dd8001a70 con 0x7f9de00d0070 2024-05-09T05:01:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.182+0000 7f9dde7fc640 1 -- 172.21.15.60:0/29709123 >> v1:172.21.15.60:6789/0 conn(0x7f9de00d0070 legacy=0x7f9de0163210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.182+0000 7f9dde7fc640 1 --2- 172.21.15.60:0/29709123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dcc003a00 0x7f9dcc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.182+0000 7f9dde7fc640 1 --2- 172.21.15.60:0/29709123 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9de0098af0 0x7f9dcc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.182+0000 7f9dde7fc640 1 --2- 172.21.15.60:0/29709123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dcc004d40 0x7f9dcc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.182+0000 7f9dde7fc640 1 -- 172.21.15.60:0/29709123 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9de00ae9b0 con 0x7f9dcc004d40 2024-05-09T05:01:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.182+0000 7f9dde7fc640 1 -- 172.21.15.60:0/29709123 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9de00aec90 con 0x7f9dcc003a00 2024-05-09T05:01:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.182+0000 7f9dde7fc640 1 -- 172.21.15.60:0/29709123 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9dcc00b9c0 con 0x7f9de0098af0 2024-05-09T05:01:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.182+0000 7f9ddf7fe640 1 --2- 172.21.15.60:0/29709123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dcc004d40 0x7f9dcc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.182+0000 7f9ddeffd640 1 --2- 172.21.15.60:0/29709123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dcc003a00 0x7f9dcc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.182+0000 7f9dd7fff640 1 --2- 172.21.15.60:0/29709123 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9de0098af0 0x7f9dcc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.182+0000 7f9ddeffd640 1 -- 172.21.15.60:0/29709123 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9de0098af0 msgr2=0x7f9dcc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.182+0000 7f9ddeffd640 1 --2- 172.21.15.60:0/29709123 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9de0098af0 0x7f9dcc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.182+0000 7f9ddeffd640 1 -- 172.21.15.60:0/29709123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dcc004d40 msgr2=0x7f9dcc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.182+0000 7f9ddeffd640 1 --2- 172.21.15.60:0/29709123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dcc004d40 0x7f9dcc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.182+0000 7f9ddeffd640 1 -- 172.21.15.60:0/29709123 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dcc003520 con 0x7f9dcc003a00 2024-05-09T05:01:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.182+0000 7f9ddf7fe640 1 --2- 172.21.15.60:0/29709123 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dcc004d40 0x7f9dcc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:01:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.183+0000 7f9ddeffd640 1 --2- 172.21.15.60:0/29709123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dcc003a00 0x7f9dcc003e20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0009ac0 tx=0x7f9dcc005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c49cf32d25d4c00 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.183+0000 7f9dde7fc640 1 -- 172.21.15.60:0/29709123 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0036070 con 0x7f9dcc003a00 2024-05-09T05:01:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.183+0000 7f9dde7fc640 1 -- 172.21.15.60:0/29709123 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dd0031400 con 0x7f9dcc003a00 2024-05-09T05:01:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.183+0000 7f9dde7fc640 1 -- 172.21.15.60:0/29709123 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0009690 con 0x7f9dcc003a00 2024-05-09T05:01:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.183+0000 7f9de5746640 1 -- 172.21.15.60:0/29709123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dcc003a00 msgr2=0x7f9dcc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.183+0000 7f9de5746640 1 --2- 172.21.15.60:0/29709123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dcc003a00 0x7f9dcc003e20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0009ac0 tx=0x7f9dcc005830 comp rx=0 tx=0).stop 2024-05-09T05:01:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.183+0000 7f9ddf7fe640 1 -- 172.21.15.60:0/29709123 reap_dead start 2024-05-09T05:01:24.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.183+0000 7f9de5746640 1 -- 172.21.15.60:0/29709123 shutdown_connections 2024-05-09T05:01:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.183+0000 7f9de5746640 1 -- 172.21.15.60:0/29709123 >> 172.21.15.60:0/29709123 conn(0x7f9de00b42e0 msgr2=0x7f9de00b46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.184+0000 7f9de5746640 1 -- 172.21.15.60:0/29709123 shutdown_connections 2024-05-09T05:01:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.184+0000 7f9de5746640 1 -- 172.21.15.60:0/29709123 wait complete. 2024-05-09T05:01:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.185+0000 7f9de5746640 1 Processor -- start 2024-05-09T05:01:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.185+0000 7f9de5746640 1 -- start start 2024-05-09T05:01:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.185+0000 7f9de5746640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dcc003a00 0x7f9de01f47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.185+0000 7f9de5746640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9de0159880 0x7f9de0159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9de5746640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9de009b460 0x7f9de009b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9de5746640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9de00b1960 con 0x7f9dcc003a00 2024-05-09T05:01:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9de5746640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9de00b17e0 con 0x7f9de009b460 2024-05-09T05:01:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9de5746640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9de00b3300 con 0x7f9de0159880 2024-05-09T05:01:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9ddeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dcc003a00 0x7f9de01f47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9ddeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dcc003a00 0x7f9de01f47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59704/0 (socket says 172.21.15.60:59704) 2024-05-09T05:01:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9ddeffd640 1 -- 172.21.15.60:0/126649921 learned_addr learned my addr 172.21.15.60:0/126649921 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9ddf7fe640 1 --2- 172.21.15.60:0/126649921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9de009b460 0x7f9de009b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9dd7fff640 1 --2- 172.21.15.60:0/126649921 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9de0159880 0x7f9de0159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9ddeffd640 1 -- 172.21.15.60:0/126649921 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9de0159880 msgr2=0x7f9de0159d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9ddeffd640 1 --2- 172.21.15.60:0/126649921 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9de0159880 0x7f9de0159d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9ddeffd640 1 -- 172.21.15.60:0/126649921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9de009b460 msgr2=0x7f9de009b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:24.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9ddeffd640 1 --2- 172.21.15.60:0/126649921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9de009b460 0x7f9de009b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9ddeffd640 1 -- 172.21.15.60:0/126649921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de015a3f0 con 0x7f9dcc003a00 2024-05-09T05:01:24.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9ddeffd640 1 --2- 172.21.15.60:0/126649921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dcc003a00 0x7f9de01f47b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0034040 tx=0x7f9dd0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:24.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9dd77fe640 1 -- 172.21.15.60:0/126649921 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0040020 con 0x7f9dcc003a00 2024-05-09T05:01:24.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9dd77fe640 1 -- 172.21.15.60:0/126649921 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dd0003940 con 0x7f9dcc003a00 2024-05-09T05:01:24.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9de5746640 1 -- 172.21.15.60:0/126649921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9de015a6d0 con 0x7f9dcc003a00 2024-05-09T05:01:24.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9dd77fe640 1 -- 172.21.15.60:0/126649921 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0036040 con 0x7f9dcc003a00 2024-05-09T05:01:24.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.186+0000 7f9de5746640 1 -- 172.21.15.60:0/126649921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9de015ac90 con 0x7f9dcc003a00 2024-05-09T05:01:24.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.187+0000 7f9de5746640 1 -- 172.21.15.60:0/126649921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9de009be20 con 0x7f9dcc003a00 2024-05-09T05:01:24.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.188+0000 7f9dd77fe640 1 -- 172.21.15.60:0/126649921 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9dd0043020 con 0x7f9dcc003a00 2024-05-09T05:01:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.188+0000 7f9dd77fe640 1 --2- 172.21.15.60:0/126649921 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9dc0076300 0x7f9dc00787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.189+0000 7f9dd77fe640 1 -- 172.21.15.60:0/126649921 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6496+0+0 (secure 0 0 0) 0x7f9dd00b7a30 con 0x7f9dcc003a00 2024-05-09T05:01:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.189+0000 7f9dd7fff640 1 --2- 172.21.15.60:0/126649921 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9dc0076300 0x7f9dc00787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.189+0000 7f9dd7fff640 1 --2- 172.21.15.60:0/126649921 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9dc0076300 0x7f9dc00787c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9de009c430 tx=0x7f9dc8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:24.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.193+0000 7f9dd77fe640 1 -- 172.21.15.60:0/126649921 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9dd004a070 con 0x7f9dcc003a00 2024-05-09T05:01:24.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.421+0000 7f9de5746640 1 -- 172.21.15.60:0/126649921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9de0051040 con 0x7f9dcc003a00 2024-05-09T05:01:24.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.421+0000 7f9dd77fe640 1 -- 172.21.15.60:0/126649921 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5183 (secure 0 0 0) 0x7f9dd00810c0 con 0x7f9dcc003a00 2024-05-09T05:01:24.422 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:24.422 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4547,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2372252858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2372252858},{"type":"v1","addr":"172.21.15.114:6836","nonce":2372252858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1761282610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1761282610},{"type":"v1","addr":"172.21.15.114:6837","nonce":1761282610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4765,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/300408012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":300408012},{"type":"v1","addr":"172.21.15.60:6835","nonce":300408012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:04.698975+0000","modified":"2024-05-09T05:01:18.144520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4893},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4893":{"gid":4893,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/399160914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":399160914},{"type":"v1","addr":"172.21.15.60:6834","nonce":399160914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4893,"qdb_cluster":[4893]},"id":2}]} 2024-05-09T05:01:24.423 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 18 2024-05-09T05:01:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.423+0000 7f9de5746640 1 -- 172.21.15.60:0/126649921 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9dc0076300 msgr2=0x7f9dc00787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.423+0000 7f9de5746640 1 --2- 172.21.15.60:0/126649921 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9dc0076300 0x7f9dc00787c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9de009c430 tx=0x7f9dc8009190 comp rx=0 tx=0).stop 2024-05-09T05:01:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.423+0000 7f9de5746640 1 -- 172.21.15.60:0/126649921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dcc003a00 msgr2=0x7f9de01f47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.423+0000 7f9de5746640 1 --2- 172.21.15.60:0/126649921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dcc003a00 0x7f9de01f47b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0034040 tx=0x7f9dd0004730 comp rx=0 tx=0).stop 2024-05-09T05:01:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.424+0000 7f9de5746640 1 -- 172.21.15.60:0/126649921 shutdown_connections 2024-05-09T05:01:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.424+0000 7f9de5746640 1 --2- 172.21.15.60:0/126649921 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9dc0076300 0x7f9dc00787c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.424+0000 7f9de5746640 1 --2- 172.21.15.60:0/126649921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9de009b460 0x7f9de009b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.424+0000 7f9de5746640 1 --2- 172.21.15.60:0/126649921 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9de0159880 0x7f9de0159d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.424+0000 7f9de5746640 1 --2- 172.21.15.60:0/126649921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dcc003a00 0x7f9de01f47b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.424+0000 7f9de5746640 1 -- 172.21.15.60:0/126649921 >> 172.21.15.60:0/126649921 conn(0x7f9de00b42e0 msgr2=0x7f9de00b2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.424+0000 7f9de5746640 1 -- 172.21.15.60:0/126649921 shutdown_connections 2024-05-09T05:01:24.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.424+0000 7f9de5746640 1 -- 172.21.15.60:0/126649921 wait complete. 2024-05-09T05:01:24.436 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:24.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.539+0000 7fb6d4aa0640 1 Processor -- start 2024-05-09T05:01:24.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.539+0000 7fb6d4aa0640 1 -- start start 2024-05-09T05:01:24.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.540+0000 7fb6d4aa0640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d015f300 0x7fb6d0072220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.540+0000 7fb6d4aa0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb6d0073de0 0x7fb6d00702a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.540+0000 7fb6d4aa0640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d0072760 con 0x7fb6d006fe20 2024-05-09T05:01:24.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.540+0000 7fb6d4aa0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fb6d0073240 con 0x7fb6d0073de0 2024-05-09T05:01:24.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.540+0000 7fb6d4aa0640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb6d00d08e0 con 0x7fb6d015f300 2024-05-09T05:01:24.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.540+0000 7fb6ced9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb6d0073de0 0x7fb6d00702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.540+0000 7fb6ced9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb6d0073de0 0x7fb6d00702a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59706/0 (socket says 172.21.15.60:59706) 2024-05-09T05:01:24.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.540+0000 7fb6ced9d640 1 -- 172.21.15.60:0/3861658830 learned_addr learned my addr 172.21.15.60:0/3861658830 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:24.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.540+0000 7fb6ce59c640 1 --2- 172.21.15.60:0/3861658830 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d015f300 0x7fb6d0072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.540+0000 7fb6ced9d640 1 -- 172.21.15.60:0/3861658830 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d015f300 msgr2=0x7fb6d0072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:24.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.540+0000 7fb6ced9d640 1 --2- 172.21.15.60:0/3861658830 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d015f300 0x7fb6d0072220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.540+0000 7fb6ced9d640 1 -- 172.21.15.60:0/3861658830 >> v1:172.21.15.114:6789/0 conn(0x7fb6d006fe20 legacy=0x7fb6d00736d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:24.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.540+0000 7fb6ced9d640 1 -- 172.21.15.60:0/3861658830 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d00d0bc0 con 0x7fb6d0073de0 2024-05-09T05:01:24.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.541+0000 7fb6ced9d640 1 --2- 172.21.15.60:0/3861658830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d0073de0 0x7fb6d00702a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8009e50 tx=0x7fb6c802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c547f05ed6cf31aa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:24.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.541+0000 7fb6cdd9b640 1 -- 172.21.15.60:0/3861658830 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8036070 con 0x7fb6d0073de0 2024-05-09T05:01:24.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.541+0000 7fb6cdd9b640 1 -- 172.21.15.60:0/3861658830 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6c8031600 con 0x7fb6d0073de0 2024-05-09T05:01:24.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.541+0000 7fb6cdd9b640 1 -- 172.21.15.60:0/3861658830 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c80319d0 con 0x7fb6d0073de0 2024-05-09T05:01:24.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.541+0000 7fb6d4aa0640 1 -- 172.21.15.60:0/3861658830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d0073de0 msgr2=0x7fb6d00702a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:24.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.541+0000 7fb6d4aa0640 1 --2- 172.21.15.60:0/3861658830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d0073de0 0x7fb6d00702a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8009e50 tx=0x7fb6c802a680 comp rx=0 tx=0).stop 2024-05-09T05:01:24.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.542+0000 7fb6d4aa0640 1 -- 172.21.15.60:0/3861658830 shutdown_connections 2024-05-09T05:01:24.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.542+0000 7fb6d4aa0640 1 --2- 172.21.15.60:0/3861658830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d0073de0 0x7fb6d00702a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.542+0000 7fb6d4aa0640 1 --2- 172.21.15.60:0/3861658830 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d015f300 0x7fb6d0072220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.542+0000 7fb6d4aa0640 1 -- 172.21.15.60:0/3861658830 >> 172.21.15.60:0/3861658830 conn(0x7fb6d00b4ef0 msgr2=0x7fb6d006e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:24.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.542+0000 7fb6d4aa0640 1 -- 172.21.15.60:0/3861658830 shutdown_connections 2024-05-09T05:01:24.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.542+0000 7fb6d4aa0640 1 -- 172.21.15.60:0/3861658830 wait complete. 2024-05-09T05:01:24.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.543+0000 7fb6d4aa0640 1 Processor -- start 2024-05-09T05:01:24.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.543+0000 7fb6d4aa0640 1 -- start start 2024-05-09T05:01:24.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.544+0000 7fb6d4aa0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6d006fe20 0x7fb6d015c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.544+0000 7fb6d4aa0640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d0073de0 0x7fb6d00d2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.544+0000 7fb6d4aa0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d015f300 0x7fb6d00d3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.544+0000 7fb6d4aa0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d00d04f0 con 0x7fb6d015f300 2024-05-09T05:01:24.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.544+0000 7fb6d4aa0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d009c1e0 con 0x7fb6d006fe20 2024-05-09T05:01:24.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.544+0000 7fb6d4aa0640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb6d00d0670 con 0x7fb6d0073de0 2024-05-09T05:01:24.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.544+0000 7fb6ced9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d015f300 0x7fb6d00d3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.544+0000 7fb6ced9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d015f300 0x7fb6d00d3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59710/0 (socket says 172.21.15.60:59710) 2024-05-09T05:01:24.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.544+0000 7fb6ced9d640 1 -- 172.21.15.60:0/3923388220 learned_addr learned my addr 172.21.15.60:0/3923388220 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:24.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.544+0000 7fb6ce59c640 1 --2- 172.21.15.60:0/3923388220 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6d006fe20 0x7fb6d015c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.544+0000 7fb6c5d9b640 1 --2- 172.21.15.60:0/3923388220 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d0073de0 0x7fb6d00d2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.544+0000 7fb6ced9d640 1 -- 172.21.15.60:0/3923388220 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d0073de0 msgr2=0x7fb6d00d2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:24.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.544+0000 7fb6ced9d640 1 --2- 172.21.15.60:0/3923388220 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d0073de0 0x7fb6d00d2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.544+0000 7fb6ced9d640 1 -- 172.21.15.60:0/3923388220 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6d006fe20 msgr2=0x7fb6d015c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:24.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.544+0000 7fb6ced9d640 1 --2- 172.21.15.60:0/3923388220 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6d006fe20 0x7fb6d015c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.544+0000 7fb6ced9d640 1 -- 172.21.15.60:0/3923388220 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d00d24b0 con 0x7fb6d015f300 2024-05-09T05:01:24.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.544+0000 7fb6ced9d640 1 --2- 172.21.15.60:0/3923388220 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d015f300 0x7fb6d00d3530 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8009e50 tx=0x7fb6c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:24.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.545+0000 7fb6c77fe640 1 -- 172.21.15.60:0/3923388220 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8036050 con 0x7fb6d015f300 2024-05-09T05:01:24.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.545+0000 7fb6c77fe640 1 -- 172.21.15.60:0/3923388220 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6c8042070 con 0x7fb6d015f300 2024-05-09T05:01:24.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.545+0000 7fb6d4aa0640 1 -- 172.21.15.60:0/3923388220 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6d00d2790 con 0x7fb6d015f300 2024-05-09T05:01:24.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.545+0000 7fb6c77fe640 1 -- 172.21.15.60:0/3923388220 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8009510 con 0x7fb6d015f300 2024-05-09T05:01:24.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.545+0000 7fb6d4aa0640 1 -- 172.21.15.60:0/3923388220 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6d0161200 con 0x7fb6d015f300 2024-05-09T05:01:24.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.546+0000 7fb6d4aa0640 1 -- 172.21.15.60:0/3923388220 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6d00d2960 con 0x7fb6d015f300 2024-05-09T05:01:24.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.546+0000 7fb6c77fe640 1 -- 172.21.15.60:0/3923388220 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb6c8007bf0 con 0x7fb6d015f300 2024-05-09T05:01:24.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.547+0000 7fb6c77fe640 1 --2- 172.21.15.60:0/3923388220 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb6b00762d0 0x7fb6b0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.547+0000 7fb6c77fe640 1 -- 172.21.15.60:0/3923388220 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6496+0+0 (secure 0 0 0) 0x7fb6c80b85e0 con 0x7fb6d015f300 2024-05-09T05:01:24.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.547+0000 7fb6ce59c640 1 --2- 172.21.15.60:0/3923388220 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb6b00762d0 0x7fb6b0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.548+0000 7fb6ce59c640 1 --2- 172.21.15.60:0/3923388220 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb6b00762d0 0x7fb6b0078790 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb6d00b1ce0 tx=0x7fb6c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:24.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.551+0000 7fb6c77fe640 1 -- 172.21.15.60:0/3923388220 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb6d0072760 con 0x7fb6d015f300 2024-05-09T05:01:24.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.784+0000 7fb6d4aa0640 1 -- 172.21.15.60:0/3923388220 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6d0161400 con 0x7fb6d015f300 2024-05-09T05:01:24.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.784+0000 7fb6c77fe640 1 -- 172.21.15.60:0/3923388220 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5183 (secure 0 0 0) 0x7fb6c8081670 con 0x7fb6d015f300 2024-05-09T05:01:24.785 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:24.785 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4547,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2372252858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2372252858},{"type":"v1","addr":"172.21.15.114:6836","nonce":2372252858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1761282610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1761282610},{"type":"v1","addr":"172.21.15.114:6837","nonce":1761282610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4765,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/300408012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":300408012},{"type":"v1","addr":"172.21.15.60:6835","nonce":300408012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:04.698975+0000","modified":"2024-05-09T05:01:18.144520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4893},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4893":{"gid":4893,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6834/399160914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":399160914},{"type":"v1","addr":"172.21.15.60:6834","nonce":399160914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4893,"qdb_cluster":[4893]},"id":2}]} 2024-05-09T05:01:24.785 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 18 2024-05-09T05:01:24.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.786+0000 7fb6d4aa0640 1 -- 172.21.15.60:0/3923388220 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb6b00762d0 msgr2=0x7fb6b0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:24.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.786+0000 7fb6d4aa0640 1 --2- 172.21.15.60:0/3923388220 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb6b00762d0 0x7fb6b0078790 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb6d00b1ce0 tx=0x7fb6c0008040 comp rx=0 tx=0).stop 2024-05-09T05:01:24.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.786+0000 7fb6d4aa0640 1 -- 172.21.15.60:0/3923388220 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d015f300 msgr2=0x7fb6d00d3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:24.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.786+0000 7fb6d4aa0640 1 --2- 172.21.15.60:0/3923388220 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d015f300 0x7fb6d00d3530 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8009e50 tx=0x7fb6c8004a40 comp rx=0 tx=0).stop 2024-05-09T05:01:24.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.786+0000 7fb6d4aa0640 1 -- 172.21.15.60:0/3923388220 shutdown_connections 2024-05-09T05:01:24.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.786+0000 7fb6d4aa0640 1 --2- 172.21.15.60:0/3923388220 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb6b00762d0 0x7fb6b0078790 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.786+0000 7fb6d4aa0640 1 --2- 172.21.15.60:0/3923388220 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d015f300 0x7fb6d00d3530 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.786+0000 7fb6d4aa0640 1 --2- 172.21.15.60:0/3923388220 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d0073de0 0x7fb6d00d2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.786+0000 7fb6d4aa0640 1 --2- 172.21.15.60:0/3923388220 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6d006fe20 0x7fb6d015c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.786+0000 7fb6d4aa0640 1 -- 172.21.15.60:0/3923388220 >> 172.21.15.60:0/3923388220 conn(0x7fb6d00b4ef0 msgr2=0x7fb6d00b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:24.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.786+0000 7fb6d4aa0640 1 -- 172.21.15.60:0/3923388220 shutdown_connections 2024-05-09T05:01:24.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.786+0000 7fb6d4aa0640 1 -- 172.21.15.60:0/3923388220 wait complete. 2024-05-09T05:01:24.799 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:01:24.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.903+0000 7f6662075640 1 Processor -- start 2024-05-09T05:01:24.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.903+0000 7f6662075640 1 -- start start 2024-05-09T05:01:24.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.903+0000 7f6662075640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f665c06f9f0 0x7f665c06ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.903+0000 7f6662075640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f665c06ff60 con 0x7f665c1606f0 2024-05-09T05:01:24.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.904+0000 7f6662075640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f665c070240 con 0x7f665c15f300 2024-05-09T05:01:24.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.904+0000 7f6662075640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f665c070520 con 0x7f665c06f9f0 2024-05-09T05:01:24.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.904+0000 7f665bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f665c06f9f0 0x7f665c06ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.904+0000 7f6653fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f665c1606f0 0x7f665c0d33f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42154/0 (socket says 172.21.15.60:42154) 2024-05-09T05:01:24.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.904+0000 7f665bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f665c06f9f0 0x7f665c06ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59720/0 (socket says 172.21.15.60:59720) 2024-05-09T05:01:24.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.904+0000 7f6653fff640 1 -- 172.21.15.60:0/3155231311 learned_addr learned my addr 172.21.15.60:0/3155231311 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:24.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.904+0000 7f665bfff640 1 -- 172.21.15.60:0/3155231311 >> v1:172.21.15.114:6789/0 conn(0x7f665c15f300 legacy=0x7f665c1639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:24.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.904+0000 7f665bfff640 1 -- 172.21.15.60:0/3155231311 >> v1:172.21.15.60:6789/0 conn(0x7f665c1606f0 legacy=0x7f665c0d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:24.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.904+0000 7f665bfff640 1 -- 172.21.15.60:0/3155231311 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f665c06e740 con 0x7f665c06f9f0 2024-05-09T05:01:24.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.904+0000 7f665affd640 1 -- 172.21.15.60:0/3155231311 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2158177052 0 0) 0x7f665c06ff60 con 0x7f665c1606f0 2024-05-09T05:01:24.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.905+0000 7f665bfff640 1 --2- 172.21.15.60:0/3155231311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f665c06f9f0 0x7f665c06ef40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6654009dc0 tx=0x7f665402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b6d15cdffb822b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:24.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.905+0000 7f665affd640 1 -- 172.21.15.60:0/3155231311 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665402ae60 con 0x7f665c06f9f0 2024-05-09T05:01:24.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.905+0000 7f665affd640 1 -- 172.21.15.60:0/3155231311 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6654007740 con 0x7f665c06f9f0 2024-05-09T05:01:24.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.905+0000 7f665affd640 1 -- 172.21.15.60:0/3155231311 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6654007b10 con 0x7f665c06f9f0 2024-05-09T05:01:24.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.905+0000 7f6662075640 1 -- 172.21.15.60:0/3155231311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f665c06f9f0 msgr2=0x7f665c06ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:24.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.905+0000 7f6662075640 1 --2- 172.21.15.60:0/3155231311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f665c06f9f0 0x7f665c06ef40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6654009dc0 tx=0x7f665402a680 comp rx=0 tx=0).stop 2024-05-09T05:01:24.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.906+0000 7f6662075640 1 -- 172.21.15.60:0/3155231311 shutdown_connections 2024-05-09T05:01:24.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.906+0000 7f6662075640 1 --2- 172.21.15.60:0/3155231311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f665c06f9f0 0x7f665c06ef40 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.906+0000 7f6662075640 1 -- 172.21.15.60:0/3155231311 >> 172.21.15.60:0/3155231311 conn(0x7f665c09b850 msgr2=0x7f665c09bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:24.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.906+0000 7f6662075640 1 -- 172.21.15.60:0/3155231311 shutdown_connections 2024-05-09T05:01:24.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.906+0000 7f6662075640 1 -- 172.21.15.60:0/3155231311 wait complete. 2024-05-09T05:01:24.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.907+0000 7f6662075640 1 Processor -- start 2024-05-09T05:01:24.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.907+0000 7f6662075640 1 -- start start 2024-05-09T05:01:24.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.907+0000 7f6662075640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f665c06f9f0 0x7f665c09f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.907+0000 7f6662075640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f665c15f300 0x7f665c0b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.907+0000 7f6662075640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f665c1606f0 0x7f665c161c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.907+0000 7f6662075640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f665c0d3c80 con 0x7f665c15f300 2024-05-09T05:01:24.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.907+0000 7f6662075640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f665c0d3b00 con 0x7f665c1606f0 2024-05-09T05:01:24.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.907+0000 7f6662075640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f665c06f480 con 0x7f665c06f9f0 2024-05-09T05:01:24.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.908+0000 7f6653fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f665c15f300 0x7f665c0b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.908+0000 7f6653fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f665c15f300 0x7f665c0b3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59734/0 (socket says 172.21.15.60:59734) 2024-05-09T05:01:24.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.908+0000 7f6653fff640 1 -- 172.21.15.60:0/174367355 learned_addr learned my addr 172.21.15.60:0/174367355 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:24.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.908+0000 7f665bfff640 1 --2- 172.21.15.60:0/174367355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f665c1606f0 0x7f665c161c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.908+0000 7f665b7fe640 1 --2- 172.21.15.60:0/174367355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f665c06f9f0 0x7f665c09f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.908+0000 7f6653fff640 1 -- 172.21.15.60:0/174367355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f665c06f9f0 msgr2=0x7f665c09f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:24.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.908+0000 7f6653fff640 1 --2- 172.21.15.60:0/174367355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f665c06f9f0 0x7f665c09f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.908+0000 7f6653fff640 1 -- 172.21.15.60:0/174367355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f665c1606f0 msgr2=0x7f665c161c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:24.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.908+0000 7f6653fff640 1 --2- 172.21.15.60:0/174367355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f665c1606f0 0x7f665c161c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:24.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.908+0000 7f6653fff640 1 -- 172.21.15.60:0/174367355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f665c0b4250 con 0x7f665c15f300 2024-05-09T05:01:24.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.908+0000 7f6653fff640 1 --2- 172.21.15.60:0/174367355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f665c15f300 0x7f665c0b3d10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f664400c3c0 tx=0x7f664400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:24.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.908+0000 7f6658ff9640 1 -- 172.21.15.60:0/174367355 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66440074e0 con 0x7f665c15f300 2024-05-09T05:01:24.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.908+0000 7f6658ff9640 1 -- 172.21.15.60:0/174367355 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66440047e0 con 0x7f665c15f300 2024-05-09T05:01:24.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.908+0000 7f6662075640 1 -- 172.21.15.60:0/174367355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f665c0b2c70 con 0x7f665c15f300 2024-05-09T05:01:24.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.908+0000 7f6658ff9640 1 -- 172.21.15.60:0/174367355 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6644004c10 con 0x7f665c15f300 2024-05-09T05:01:24.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.908+0000 7f6662075640 1 -- 172.21.15.60:0/174367355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f665c0b3230 con 0x7f665c15f300 2024-05-09T05:01:24.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.909+0000 7f6662075640 1 -- 172.21.15.60:0/174367355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f665c072020 con 0x7f665c15f300 2024-05-09T05:01:24.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.910+0000 7f6658ff9640 1 -- 172.21.15.60:0/174367355 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6644011070 con 0x7f665c15f300 2024-05-09T05:01:24.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.911+0000 7f6658ff9640 1 --2- 172.21.15.60:0/174367355 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f663c076300 0x7f663c0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:24.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.911+0000 7f6658ff9640 1 -- 172.21.15.60:0/174367355 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6496+0+0 (secure 0 0 0) 0x7f66440973b0 con 0x7f665c15f300 2024-05-09T05:01:24.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.911+0000 7f665b7fe640 1 --2- 172.21.15.60:0/174367355 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f663c076300 0x7f663c0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:24.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.911+0000 7f665b7fe640 1 --2- 172.21.15.60:0/174367355 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f663c076300 0x7f663c0787c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f664c004ea0 tx=0x7f664c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:24.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:24.915+0000 7f6658ff9640 1 -- 172.21.15.60:0/174367355 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f665c072020 con 0x7f665c15f300 2024-05-09T05:01:25.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.106+0000 7f6662075640 1 -- 172.21.15.60:0/174367355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f665c051040 con 0x7f665c15f300 2024-05-09T05:01:25.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.108+0000 7f6658ff9640 1 -- 172.21.15.60:0/174367355 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+18730 (secure 0 0 0) 0x7f6644060a90 con 0x7f665c15f300 2024-05-09T05:01:25.108 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:25.109 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":39,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:01:19.767056+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:01:02.773384+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-05-09T05:01:03.658182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-05-09T05:01:16.847013+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:01:25.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.110+0000 7f6662075640 1 -- 172.21.15.60:0/174367355 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f663c076300 msgr2=0x7f663c0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:25.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.110+0000 7f6662075640 1 --2- 172.21.15.60:0/174367355 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f663c076300 0x7f663c0787c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f664c004ea0 tx=0x7f664c009040 comp rx=0 tx=0).stop 2024-05-09T05:01:25.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.110+0000 7f6662075640 1 -- 172.21.15.60:0/174367355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f665c15f300 msgr2=0x7f665c0b3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:25.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.110+0000 7f6662075640 1 --2- 172.21.15.60:0/174367355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f665c15f300 0x7f665c0b3d10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f664400c3c0 tx=0x7f664400c880 comp rx=0 tx=0).stop 2024-05-09T05:01:25.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.110+0000 7f6662075640 1 -- 172.21.15.60:0/174367355 shutdown_connections 2024-05-09T05:01:25.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.110+0000 7f6662075640 1 --2- 172.21.15.60:0/174367355 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f663c076300 0x7f663c0787c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:25.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.110+0000 7f6662075640 1 --2- 172.21.15.60:0/174367355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f665c1606f0 0x7f665c161c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:25.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.110+0000 7f6662075640 1 --2- 172.21.15.60:0/174367355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f665c15f300 0x7f665c0b3d10 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:25.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.110+0000 7f6662075640 1 --2- 172.21.15.60:0/174367355 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f665c06f9f0 0x7f665c09f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:25.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.110+0000 7f6662075640 1 -- 172.21.15.60:0/174367355 >> 172.21.15.60:0/174367355 conn(0x7f665c09b850 msgr2=0x7f665c0d2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:25.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.110+0000 7f6662075640 1 -- 172.21.15.60:0/174367355 shutdown_connections 2024-05-09T05:01:25.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.110+0000 7f6662075640 1 -- 172.21.15.60:0/174367355 wait complete. 2024-05-09T05:01:25.122 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T05:01:25.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.226+0000 7f522919d640 1 Processor -- start 2024-05-09T05:01:25.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.226+0000 7f522919d640 1 -- start start 2024-05-09T05:01:25.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.227+0000 7f522919d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f522406fcb0 0x7f52240d2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:25.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.227+0000 7f522919d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f52240d2650 0x7f52240d1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:25.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.227+0000 7f522919d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52240b31d0 con 0x7f522406e7f0 2024-05-09T05:01:25.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.227+0000 7f522919d640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f52240b2f80 con 0x7f52240d2650 2024-05-09T05:01:25.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.227+0000 7f522919d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f52240b34b0 con 0x7f522406fcb0 2024-05-09T05:01:25.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.227+0000 7f5222d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f522406e7f0 0x7f52240d1050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42162/0 (socket says 172.21.15.60:42162) 2024-05-09T05:01:25.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.227+0000 7f5222d9d640 1 -- 172.21.15.60:0/1558712994 learned_addr learned my addr 172.21.15.60:0/1558712994 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:25.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.227+0000 7f522359e640 1 --2- 172.21.15.60:0/1558712994 >> v2:172.21.15.114:3300/0 conn(0x7f52240d2650 0x7f52240d1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:25.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.227+0000 7f522259c640 1 --2- 172.21.15.60:0/1558712994 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f522406fcb0 0x7f52240d2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:25.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f5221d9b640 1 -- 172.21.15.60:0/1558712994 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2329995561 0 0) 0x7f52240b31d0 con 0x7f522406e7f0 2024-05-09T05:01:25.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f5221d9b640 1 -- 172.21.15.60:0/1558712994 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5210003520 con 0x7f522406e7f0 2024-05-09T05:01:25.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f5221d9b640 1 -- 172.21.15.60:0/1558712994 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2737905259 0 0) 0x7f5210003520 con 0x7f522406e7f0 2024-05-09T05:01:25.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f5221d9b640 1 -- 172.21.15.60:0/1558712994 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f522406fcb0 msgr2=0x7f52240d2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:25.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f5221d9b640 1 --2- 172.21.15.60:0/1558712994 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f522406fcb0 0x7f52240d2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:25.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f5221d9b640 1 -- 172.21.15.60:0/1558712994 >> v2:172.21.15.114:3300/0 conn(0x7f52240d2650 msgr2=0x7f52240d1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:25.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f5221d9b640 1 --2- 172.21.15.60:0/1558712994 >> v2:172.21.15.114:3300/0 conn(0x7f52240d2650 0x7f52240d1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:25.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f5221d9b640 1 -- 172.21.15.60:0/1558712994 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52240b3790 con 0x7f522406e7f0 2024-05-09T05:01:25.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f5221d9b640 1 -- 172.21.15.60:0/1558712994 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f5214001a70 con 0x7f522406e7f0 2024-05-09T05:01:25.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f5221d9b640 1 -- 172.21.15.60:0/1558712994 >> v1:172.21.15.60:6789/0 conn(0x7f522406e7f0 legacy=0x7f52240d1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:25.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f5221d9b640 1 --2- 172.21.15.60:0/1558712994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5210003a00 0x7f5210003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:25.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f5221d9b640 1 --2- 172.21.15.60:0/1558712994 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f522406fcb0 0x7f5210004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:25.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f5221d9b640 1 --2- 172.21.15.60:0/1558712994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5210004d40 0x7f52100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:25.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f5221d9b640 1 -- 172.21.15.60:0/1558712994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f52240b2f80 con 0x7f5210004d40 2024-05-09T05:01:25.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f5221d9b640 1 -- 172.21.15.60:0/1558712994 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f52240b34b0 con 0x7f5210003a00 2024-05-09T05:01:25.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f5221d9b640 1 -- 172.21.15.60:0/1558712994 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f521000b9f0 con 0x7f522406fcb0 2024-05-09T05:01:25.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f522359e640 1 --2- 172.21.15.60:0/1558712994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5210004d40 0x7f52100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:25.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f5222d9d640 1 --2- 172.21.15.60:0/1558712994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5210003a00 0x7f5210003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:25.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f522259c640 1 --2- 172.21.15.60:0/1558712994 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f522406fcb0 0x7f5210004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:25.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f522359e640 1 -- 172.21.15.60:0/1558712994 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f522406fcb0 msgr2=0x7f5210004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:25.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f522359e640 1 --2- 172.21.15.60:0/1558712994 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f522406fcb0 0x7f5210004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:25.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f522359e640 1 -- 172.21.15.60:0/1558712994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5210003a00 msgr2=0x7f5210003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:25.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f522359e640 1 --2- 172.21.15.60:0/1558712994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5210003a00 0x7f5210003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:25.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.228+0000 7f522359e640 1 -- 172.21.15.60:0/1558712994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5210003520 con 0x7f5210004d40 2024-05-09T05:01:25.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.229+0000 7f522359e640 1 --2- 172.21.15.60:0/1558712994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5210004d40 0x7f52100051c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f521c009a90 tx=0x7f5210005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b29e40b91ce024e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:25.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.229+0000 7f5221d9b640 1 -- 172.21.15.60:0/1558712994 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c036070 con 0x7f5210004d40 2024-05-09T05:01:25.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.229+0000 7f5221d9b640 1 -- 172.21.15.60:0/1558712994 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f521c031400 con 0x7f5210004d40 2024-05-09T05:01:25.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.229+0000 7f5221d9b640 1 -- 172.21.15.60:0/1558712994 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c009690 con 0x7f5210004d40 2024-05-09T05:01:25.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.230+0000 7f522919d640 1 -- 172.21.15.60:0/1558712994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5210004d40 msgr2=0x7f52100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.230+0000 7f522919d640 1 --2- 172.21.15.60:0/1558712994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5210004d40 0x7f52100051c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f521c009a90 tx=0x7f5210005830 comp rx=0 tx=0).stop 2024-05-09T05:01:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.230+0000 7f522359e640 1 -- 172.21.15.60:0/1558712994 reap_dead start 2024-05-09T05:01:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.230+0000 7f522919d640 1 -- 172.21.15.60:0/1558712994 shutdown_connections 2024-05-09T05:01:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.230+0000 7f522919d640 1 -- 172.21.15.60:0/1558712994 >> 172.21.15.60:0/1558712994 conn(0x7f52240993f0 msgr2=0x7f5224099800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.230+0000 7f522919d640 1 -- 172.21.15.60:0/1558712994 shutdown_connections 2024-05-09T05:01:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.230+0000 7f522919d640 1 -- 172.21.15.60:0/1558712994 wait complete. 2024-05-09T05:01:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.231+0000 7f522919d640 1 Processor -- start 2024-05-09T05:01:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.231+0000 7f522919d640 1 -- start start 2024-05-09T05:01:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.232+0000 7f522919d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5210004d40 0x7f52240b1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.232+0000 7f522919d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f522415ae00 0x7f522415b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.232+0000 7f522919d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f522406d850 0x7f522406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.232+0000 7f522919d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f52240d1ca0 con 0x7f5210004d40 2024-05-09T05:01:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.232+0000 7f522919d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f52240b3b30 con 0x7f522406d850 2024-05-09T05:01:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.232+0000 7f522919d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f52240d1e20 con 0x7f522415ae00 2024-05-09T05:01:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.232+0000 7f5222d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5210004d40 0x7f52240b1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.232+0000 7f5222d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5210004d40 0x7f52240b1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59752/0 (socket says 172.21.15.60:59752) 2024-05-09T05:01:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.232+0000 7f5222d9d640 1 -- 172.21.15.60:0/3188802068 learned_addr learned my addr 172.21.15.60:0/3188802068 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.232+0000 7f522359e640 1 --2- 172.21.15.60:0/3188802068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f522406d850 0x7f522406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.232+0000 7f522259c640 1 --2- 172.21.15.60:0/3188802068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f522415ae00 0x7f522415b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.233+0000 7f5222d9d640 1 -- 172.21.15.60:0/3188802068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f522415ae00 msgr2=0x7f522415b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.233+0000 7f5222d9d640 1 --2- 172.21.15.60:0/3188802068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f522415ae00 0x7f522415b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.233+0000 7f5222d9d640 1 -- 172.21.15.60:0/3188802068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f522406d850 msgr2=0x7f522406dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.233+0000 7f5222d9d640 1 --2- 172.21.15.60:0/3188802068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f522406d850 0x7f522406dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.233+0000 7f5222d9d640 1 -- 172.21.15.60:0/3188802068 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522415b7c0 con 0x7f5210004d40 2024-05-09T05:01:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.233+0000 7f5222d9d640 1 --2- 172.21.15.60:0/3188802068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5210004d40 0x7f52240b1ae0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5214007a80 tx=0x7f521400a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.233+0000 7f521b7fe640 1 -- 172.21.15.60:0/3188802068 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5214007ab0 con 0x7f5210004d40 2024-05-09T05:01:25.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.233+0000 7f521b7fe640 1 -- 172.21.15.60:0/3188802068 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5214018070 con 0x7f5210004d40 2024-05-09T05:01:25.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.233+0000 7f522919d640 1 -- 172.21.15.60:0/3188802068 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f522406c410 con 0x7f5210004d40 2024-05-09T05:01:25.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.233+0000 7f521b7fe640 1 -- 172.21.15.60:0/3188802068 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5214013630 con 0x7f5210004d40 2024-05-09T05:01:25.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.233+0000 7f522919d640 1 -- 172.21.15.60:0/3188802068 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f522406c990 con 0x7f5210004d40 2024-05-09T05:01:25.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.234+0000 7f522919d640 1 -- 172.21.15.60:0/3188802068 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5224051040 con 0x7f5210004d40 2024-05-09T05:01:25.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.235+0000 7f521b7fe640 1 -- 172.21.15.60:0/3188802068 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f52140137d0 con 0x7f5210004d40 2024-05-09T05:01:25.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.235+0000 7f521b7fe640 1 --2- 172.21.15.60:0/3188802068 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f52040762d0 0x7f5204078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:25.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.235+0000 7f521b7fe640 1 -- 172.21.15.60:0/3188802068 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6496+0+0 (secure 0 0 0) 0x7f5214098000 con 0x7f5210004d40 2024-05-09T05:01:25.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.235+0000 7f522259c640 1 --2- 172.21.15.60:0/3188802068 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f52040762d0 0x7f5204078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:25.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.236+0000 7f522259c640 1 --2- 172.21.15.60:0/3188802068 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f52040762d0 0x7f5204078790 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f52240b2950 tx=0x7f520c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:25.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.239+0000 7f521b7fe640 1 -- 172.21.15.60:0/3188802068 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f522415b7c0 con 0x7f5210004d40 2024-05-09T05:01:25.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.469+0000 7f522919d640 1 -- 172.21.15.60:0/3188802068 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f522406f110 con 0x7f5210004d40 2024-05-09T05:01:25.470 INFO:tasks.ceph.mon.a.smithi060.stderr:2024-05-09T05:01:25.470+0000 7fde6feeb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T05:01:25.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.472+0000 7f521b7fe640 1 -- 172.21.15.60:0/3188802068 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v19) ==== 161+0+0 (secure 0 0 0) 0x7f5214061650 con 0x7f5210004d40 2024-05-09T05:01:25.472 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T05:01:25.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.475+0000 7f522919d640 1 -- 172.21.15.60:0/3188802068 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f52040762d0 msgr2=0x7f5204078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:25.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.475+0000 7f522919d640 1 --2- 172.21.15.60:0/3188802068 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f52040762d0 0x7f5204078790 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f52240b2950 tx=0x7f520c009190 comp rx=0 tx=0).stop 2024-05-09T05:01:25.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.475+0000 7f522919d640 1 -- 172.21.15.60:0/3188802068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5210004d40 msgr2=0x7f52240b1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:25.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.475+0000 7f522919d640 1 --2- 172.21.15.60:0/3188802068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5210004d40 0x7f52240b1ae0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5214007a80 tx=0x7f521400a730 comp rx=0 tx=0).stop 2024-05-09T05:01:25.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.475+0000 7f522919d640 1 -- 172.21.15.60:0/3188802068 shutdown_connections 2024-05-09T05:01:25.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.475+0000 7f522919d640 1 --2- 172.21.15.60:0/3188802068 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f52040762d0 0x7f5204078790 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:25.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.475+0000 7f522919d640 1 --2- 172.21.15.60:0/3188802068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f522406d850 0x7f522406dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:25.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.475+0000 7f522919d640 1 --2- 172.21.15.60:0/3188802068 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f522415ae00 0x7f522415b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:25.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.475+0000 7f522919d640 1 --2- 172.21.15.60:0/3188802068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5210004d40 0x7f52240b1ae0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:25.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.475+0000 7f522919d640 1 -- 172.21.15.60:0/3188802068 >> 172.21.15.60:0/3188802068 conn(0x7f52240993f0 msgr2=0x7f522415da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:25.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.475+0000 7f522919d640 1 -- 172.21.15.60:0/3188802068 shutdown_connections 2024-05-09T05:01:25.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.475+0000 7f522919d640 1 -- 172.21.15.60:0/3188802068 wait complete. 2024-05-09T05:01:25.487 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-09T05:01:25.514 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2024-05-09T05:01:25.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.594+0000 7f2d9032c640 1 Processor -- start 2024-05-09T05:01:25.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.594+0000 7f2d9032c640 1 -- start start 2024-05-09T05:01:25.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.595+0000 7f2d9032c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d8815f3c0 0x7f2d8809ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:25.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.595+0000 7f2d9032c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2d88073fb0 0x7f2d880b4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:25.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.595+0000 7f2d9032c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d8815f7a0 con 0x7f2d880b4b60 2024-05-09T05:01:25.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.595+0000 7f2d9032c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f2d88071cb0 con 0x7f2d88073fb0 2024-05-09T05:01:25.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.595+0000 7f2d9032c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f2d8809f480 con 0x7f2d8815f3c0 2024-05-09T05:01:25.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.595+0000 7f2d8d8c7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2d88073fb0 0x7f2d880b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:25.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.595+0000 7f2d8d8c7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2d88073fb0 0x7f2d880b4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59770/0 (socket says 172.21.15.60:59770) 2024-05-09T05:01:25.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.595+0000 7f2d8d8c7640 1 -- 172.21.15.60:0/2915654625 learned_addr learned my addr 172.21.15.60:0/2915654625 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:25.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.596+0000 7f2d8d8c7640 1 -- 172.21.15.60:0/2915654625 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d8815f3c0 msgr2=0x7f2d8809ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:25.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.596+0000 7f2d8e0c8640 1 --2- 172.21.15.60:0/2915654625 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d8815f3c0 0x7f2d8809ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:25.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.596+0000 7f2d8d8c7640 1 --2- 172.21.15.60:0/2915654625 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d8815f3c0 0x7f2d8809ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:25.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.596+0000 7f2d8d8c7640 1 -- 172.21.15.60:0/2915654625 >> v1:172.21.15.114:6789/0 conn(0x7f2d880b4b60 legacy=0x7f2d8806e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:25.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.596+0000 7f2d8d8c7640 1 -- 172.21.15.60:0/2915654625 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8809f760 con 0x7f2d88073fb0 2024-05-09T05:01:25.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.596+0000 7f2d8d8c7640 1 --2- 172.21.15.60:0/2915654625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d88073fb0 0x7f2d880b4620 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009870 tx=0x7f2d7802a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88585a7d1fda7688 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:25.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.596+0000 7f2d8d0c6640 1 -- 172.21.15.60:0/2915654625 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d78037070 con 0x7f2d88073fb0 2024-05-09T05:01:25.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.597+0000 7f2d8d0c6640 1 -- 172.21.15.60:0/2915654625 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d78032440 con 0x7f2d88073fb0 2024-05-09T05:01:25.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.597+0000 7f2d8d0c6640 1 -- 172.21.15.60:0/2915654625 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d78032810 con 0x7f2d88073fb0 2024-05-09T05:01:25.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.597+0000 7f2d9032c640 1 -- 172.21.15.60:0/2915654625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d88073fb0 msgr2=0x7f2d880b4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:25.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.597+0000 7f2d9032c640 1 --2- 172.21.15.60:0/2915654625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d88073fb0 0x7f2d880b4620 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009870 tx=0x7f2d7802a130 comp rx=0 tx=0).stop 2024-05-09T05:01:25.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.597+0000 7f2d9032c640 1 -- 172.21.15.60:0/2915654625 shutdown_connections 2024-05-09T05:01:25.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.597+0000 7f2d9032c640 1 --2- 172.21.15.60:0/2915654625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d88073fb0 0x7f2d880b4620 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:25.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.597+0000 7f2d9032c640 1 --2- 172.21.15.60:0/2915654625 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d8815f3c0 0x7f2d8809ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:25.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.597+0000 7f2d9032c640 1 -- 172.21.15.60:0/2915654625 >> 172.21.15.60:0/2915654625 conn(0x7f2d880d1470 msgr2=0x7f2d880d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:25.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.597+0000 7f2d9032c640 1 -- 172.21.15.60:0/2915654625 shutdown_connections 2024-05-09T05:01:25.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.597+0000 7f2d9032c640 1 -- 172.21.15.60:0/2915654625 wait complete. 2024-05-09T05:01:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.598+0000 7f2d9032c640 1 Processor -- start 2024-05-09T05:01:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.598+0000 7f2d9032c640 1 -- start start 2024-05-09T05:01:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.599+0000 7f2d9032c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d88073fb0 0x7f2d880d0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.599+0000 7f2d9032c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d880b4b60 0x7f2d880cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.599+0000 7f2d9032c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d8815f3c0 0x7f2d880cb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.599+0000 7f2d9032c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2d88073150 con 0x7f2d8815f3c0 2024-05-09T05:01:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.599+0000 7f2d9032c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8809f120 con 0x7f2d880b4b60 2024-05-09T05:01:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.599+0000 7f2d9032c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f2d880732d0 con 0x7f2d88073fb0 2024-05-09T05:01:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.599+0000 7f2d8e8c9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d8815f3c0 0x7f2d880cb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.599+0000 7f2d8e8c9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d8815f3c0 0x7f2d880cb310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59778/0 (socket says 172.21.15.60:59778) 2024-05-09T05:01:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.599+0000 7f2d8e8c9640 1 -- 172.21.15.60:0/2240526658 learned_addr learned my addr 172.21.15.60:0/2240526658 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.599+0000 7f2d8d8c7640 1 --2- 172.21.15.60:0/2240526658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d880b4b60 0x7f2d880cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.599+0000 7f2d8e0c8640 1 --2- 172.21.15.60:0/2240526658 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d88073fb0 0x7f2d880d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.600+0000 7f2d8e8c9640 1 -- 172.21.15.60:0/2240526658 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d88073fb0 msgr2=0x7f2d880d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.600+0000 7f2d8e8c9640 1 --2- 172.21.15.60:0/2240526658 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d88073fb0 0x7f2d880d0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.600+0000 7f2d8e8c9640 1 -- 172.21.15.60:0/2240526658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d880b4b60 msgr2=0x7f2d880cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.600+0000 7f2d8e8c9640 1 --2- 172.21.15.60:0/2240526658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d880b4b60 0x7f2d880cadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.600+0000 7f2d8e8c9640 1 -- 172.21.15.60:0/2240526658 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d880d3f50 con 0x7f2d8815f3c0 2024-05-09T05:01:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.600+0000 7f2d8e8c9640 1 --2- 172.21.15.60:0/2240526658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d8815f3c0 0x7f2d880cb310 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2d8000c780 tx=0x7f2d8000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.600+0000 7f2d7effd640 1 -- 172.21.15.60:0/2240526658 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d80004b60 con 0x7f2d8815f3c0 2024-05-09T05:01:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.600+0000 7f2d7effd640 1 -- 172.21.15.60:0/2240526658 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d80004d00 con 0x7f2d8815f3c0 2024-05-09T05:01:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.600+0000 7f2d7effd640 1 -- 172.21.15.60:0/2240526658 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8000aa90 con 0x7f2d8815f3c0 2024-05-09T05:01:25.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.600+0000 7f2d9032c640 1 -- 172.21.15.60:0/2240526658 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d880d4230 con 0x7f2d8815f3c0 2024-05-09T05:01:25.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.600+0000 7f2d9032c640 1 -- 172.21.15.60:0/2240526658 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d880d47f0 con 0x7f2d8815f3c0 2024-05-09T05:01:25.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.601+0000 7f2d9032c640 1 -- 172.21.15.60:0/2240526658 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d88051040 con 0x7f2d8815f3c0 2024-05-09T05:01:25.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.602+0000 7f2d7effd640 1 -- 172.21.15.60:0/2240526658 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2d8000ac30 con 0x7f2d8815f3c0 2024-05-09T05:01:25.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.602+0000 7f2d7effd640 1 --2- 172.21.15.60:0/2240526658 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2d6c0762d0 0x7f2d6c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:25.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.602+0000 7f2d7effd640 1 -- 172.21.15.60:0/2240526658 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6582+0+0 (secure 0 0 0) 0x7f2d800979f0 con 0x7f2d8815f3c0 2024-05-09T05:01:25.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.602+0000 7f2d8e0c8640 1 --2- 172.21.15.60:0/2240526658 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2d6c0762d0 0x7f2d6c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:25.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.603+0000 7f2d8e0c8640 1 --2- 172.21.15.60:0/2240526658 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2d6c0762d0 0x7f2d6c078790 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2d8815ddf0 tx=0x7f2d84008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:25.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.607+0000 7f2d7effd640 1 -- 172.21.15.60:0/2240526658 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2d80020070 con 0x7f2d8815f3c0 2024-05-09T05:01:25.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:25.839+0000 7f2d9032c640 1 -- 172.21.15.60:0/2240526658 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f2d880700e0 con 0x7f2d8815f3c0 2024-05-09T05:01:26.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.473+0000 7f2d7effd640 1 -- 172.21.15.60:0/2240526658 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v20) ==== 104+0+0 (secure 0 0 0) 0x7f2d80060a20 con 0x7f2d8815f3c0 2024-05-09T05:01:26.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.476+0000 7f2d9032c640 1 -- 172.21.15.60:0/2240526658 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2d6c0762d0 msgr2=0x7f2d6c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:26.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.476+0000 7f2d9032c640 1 --2- 172.21.15.60:0/2240526658 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2d6c0762d0 0x7f2d6c078790 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2d8815ddf0 tx=0x7f2d84008040 comp rx=0 tx=0).stop 2024-05-09T05:01:26.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.476+0000 7f2d9032c640 1 -- 172.21.15.60:0/2240526658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d8815f3c0 msgr2=0x7f2d880cb310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:26.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.476+0000 7f2d9032c640 1 --2- 172.21.15.60:0/2240526658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d8815f3c0 0x7f2d880cb310 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2d8000c780 tx=0x7f2d8000cc40 comp rx=0 tx=0).stop 2024-05-09T05:01:26.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.476+0000 7f2d9032c640 1 -- 172.21.15.60:0/2240526658 shutdown_connections 2024-05-09T05:01:26.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.476+0000 7f2d9032c640 1 --2- 172.21.15.60:0/2240526658 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2d6c0762d0 0x7f2d6c078790 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:26.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.476+0000 7f2d9032c640 1 --2- 172.21.15.60:0/2240526658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d8815f3c0 0x7f2d880cb310 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:26.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.476+0000 7f2d9032c640 1 --2- 172.21.15.60:0/2240526658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d880b4b60 0x7f2d880cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:26.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.476+0000 7f2d9032c640 1 --2- 172.21.15.60:0/2240526658 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2d88073fb0 0x7f2d880d0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:26.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.476+0000 7f2d9032c640 1 -- 172.21.15.60:0/2240526658 >> 172.21.15.60:0/2240526658 conn(0x7f2d880d1470 msgr2=0x7f2d8809a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:26.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.476+0000 7f2d9032c640 1 -- 172.21.15.60:0/2240526658 shutdown_connections 2024-05-09T05:01:26.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.476+0000 7f2d9032c640 1 -- 172.21.15.60:0/2240526658 wait complete. 2024-05-09T05:01:26.490 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-09T05:01:26.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.643+0000 7fbdaade7640 1 Processor -- start 2024-05-09T05:01:26.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.643+0000 7fbdaade7640 1 -- start start 2024-05-09T05:01:26.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.644+0000 7fbdaade7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbda4160600 0x7fbda415d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:26.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.644+0000 7fbdaade7640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbda40b8f40 0x7fbda40b4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:26.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.644+0000 7fbdaade7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbda40b45a0 0x7fbda4160cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.644+0000 7fbdaade7640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fbda40d0f70 con 0x7fbda40b45a0 2024-05-09T05:01:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.644+0000 7fbdaade7640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fbda40d1230 con 0x7fbda40b8f40 2024-05-09T05:01:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.644+0000 7fbdaade7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbda40d1510 con 0x7fbda4160600 2024-05-09T05:01:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.644+0000 7fbda9384640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbda40b45a0 0x7fbda4160cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.644+0000 7fbda9384640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbda40b45a0 0x7fbda4160cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59782/0 (socket says 172.21.15.60:59782) 2024-05-09T05:01:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.644+0000 7fbda9384640 1 -- 172.21.15.60:0/2913190396 learned_addr learned my addr 172.21.15.60:0/2913190396 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.644+0000 7fbda3fff640 1 --2- 172.21.15.60:0/2913190396 >> v2:172.21.15.114:3300/0 conn(0x7fbda40b8f40 0x7fbda40b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.644+0000 7fbda8b83640 1 --2- 172.21.15.60:0/2913190396 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbda4160600 0x7fbda415d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.644+0000 7fbda9384640 1 -- 172.21.15.60:0/2913190396 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbda4160600 msgr2=0x7fbda415d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.644+0000 7fbda9384640 1 --2- 172.21.15.60:0/2913190396 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbda4160600 0x7fbda415d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.644+0000 7fbda9384640 1 -- 172.21.15.60:0/2913190396 >> v2:172.21.15.114:3300/0 conn(0x7fbda40b8f40 msgr2=0x7fbda40b4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.644+0000 7fbda9384640 1 --2- 172.21.15.60:0/2913190396 >> v2:172.21.15.114:3300/0 conn(0x7fbda40b8f40 0x7fbda40b4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:26.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.645+0000 7fbda9384640 1 -- 172.21.15.60:0/2913190396 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbda40d17f0 con 0x7fbda40b45a0 2024-05-09T05:01:26.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.645+0000 7fbda8b83640 1 --2- 172.21.15.60:0/2913190396 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbda4160600 0x7fbda415d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:26.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.645+0000 7fbda9384640 1 --2- 172.21.15.60:0/2913190396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbda40b45a0 0x7fbda4160cd0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbda40b4ca0 tx=0x7fbd8c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99aa3928067666ba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.645+0000 7fbda37fe640 1 -- 172.21.15.60:0/2913190396 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c037070 con 0x7fbda40b45a0 2024-05-09T05:01:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.645+0000 7fbda37fe640 1 -- 172.21.15.60:0/2913190396 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd8c032440 con 0x7fbda40b45a0 2024-05-09T05:01:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.645+0000 7fbda37fe640 1 -- 172.21.15.60:0/2913190396 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c032810 con 0x7fbda40b45a0 2024-05-09T05:01:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.646+0000 7fbdaade7640 1 -- 172.21.15.60:0/2913190396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbda40b45a0 msgr2=0x7fbda4160cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.646+0000 7fbdaade7640 1 --2- 172.21.15.60:0/2913190396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbda40b45a0 0x7fbda4160cd0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbda40b4ca0 tx=0x7fbd8c02a1e0 comp rx=0 tx=0).stop 2024-05-09T05:01:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.646+0000 7fbdaade7640 1 -- 172.21.15.60:0/2913190396 shutdown_connections 2024-05-09T05:01:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.646+0000 7fbdaade7640 1 --2- 172.21.15.60:0/2913190396 >> v2:172.21.15.114:3300/0 conn(0x7fbda40b8f40 0x7fbda40b4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.646+0000 7fbdaade7640 1 --2- 172.21.15.60:0/2913190396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbda40b45a0 0x7fbda4160cd0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.646+0000 7fbdaade7640 1 --2- 172.21.15.60:0/2913190396 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbda4160600 0x7fbda415d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:26.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.646+0000 7fbdaade7640 1 -- 172.21.15.60:0/2913190396 >> 172.21.15.60:0/2913190396 conn(0x7fbda40b3c80 msgr2=0x7fbda4070590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:26.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.646+0000 7fbdaade7640 1 -- 172.21.15.60:0/2913190396 shutdown_connections 2024-05-09T05:01:26.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.646+0000 7fbdaade7640 1 -- 172.21.15.60:0/2913190396 wait complete. 2024-05-09T05:01:26.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.647+0000 7fbdaade7640 1 Processor -- start 2024-05-09T05:01:26.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.647+0000 7fbdaade7640 1 -- start start 2024-05-09T05:01:26.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.648+0000 7fbdaade7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbda40b45a0 0x7fbda40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:26.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.648+0000 7fbdaade7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbda40b8f40 0x7fbda4145510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:26.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.648+0000 7fbdaade7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbda4160600 0x7fbda415eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:26.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.648+0000 7fbdaade7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbda41609e0 con 0x7fbda4160600 2024-05-09T05:01:26.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.648+0000 7fbdaade7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbda40d0c60 con 0x7fbda40b45a0 2024-05-09T05:01:26.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.648+0000 7fbdaade7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbda4161210 con 0x7fbda40b8f40 2024-05-09T05:01:26.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.648+0000 7fbda9384640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbda4160600 0x7fbda415eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:26.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.648+0000 7fbda9384640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbda4160600 0x7fbda415eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59784/0 (socket says 172.21.15.60:59784) 2024-05-09T05:01:26.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.648+0000 7fbda9384640 1 -- 172.21.15.60:0/491780664 learned_addr learned my addr 172.21.15.60:0/491780664 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:26.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.648+0000 7fbda8b83640 1 --2- 172.21.15.60:0/491780664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbda40b45a0 0x7fbda40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:26.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.648+0000 7fbda3fff640 1 --2- 172.21.15.60:0/491780664 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbda40b8f40 0x7fbda4145510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:26.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.649+0000 7fbda9384640 1 -- 172.21.15.60:0/491780664 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbda40b8f40 msgr2=0x7fbda4145510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:26.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.649+0000 7fbda9384640 1 --2- 172.21.15.60:0/491780664 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbda40b8f40 0x7fbda4145510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:26.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.649+0000 7fbda9384640 1 -- 172.21.15.60:0/491780664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbda40b45a0 msgr2=0x7fbda40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:26.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.649+0000 7fbda9384640 1 --2- 172.21.15.60:0/491780664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbda40b45a0 0x7fbda40cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:26.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.649+0000 7fbda9384640 1 -- 172.21.15.60:0/491780664 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbda4145fb0 con 0x7fbda4160600 2024-05-09T05:01:26.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.649+0000 7fbda8b83640 1 --2- 172.21.15.60:0/491780664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbda40b45a0 0x7fbda40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:26.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.649+0000 7fbda3fff640 1 --2- 172.21.15.60:0/491780664 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbda40b8f40 0x7fbda4145510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:26.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.649+0000 7fbda9384640 1 --2- 172.21.15.60:0/491780664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbda4160600 0x7fbda415eae0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0090b0 tx=0x7fbd8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:26.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.649+0000 7fbda17fa640 1 -- 172.21.15.60:0/491780664 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c041070 con 0x7fbda4160600 2024-05-09T05:01:26.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.649+0000 7fbda17fa640 1 -- 172.21.15.60:0/491780664 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd8c03c870 con 0x7fbda4160600 2024-05-09T05:01:26.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.649+0000 7fbdaade7640 1 -- 172.21.15.60:0/491780664 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbda4146290 con 0x7fbda4160600 2024-05-09T05:01:26.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.649+0000 7fbda17fa640 1 -- 172.21.15.60:0/491780664 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c037040 con 0x7fbda4160600 2024-05-09T05:01:26.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.649+0000 7fbdaade7640 1 -- 172.21.15.60:0/491780664 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbda40ce7e0 con 0x7fbda4160600 2024-05-09T05:01:26.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.650+0000 7fbdaade7640 1 -- 172.21.15.60:0/491780664 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbda4051040 con 0x7fbda4160600 2024-05-09T05:01:26.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.651+0000 7fbda17fa640 1 -- 172.21.15.60:0/491780664 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbd8c044070 con 0x7fbda4160600 2024-05-09T05:01:26.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.651+0000 7fbda17fa640 1 --2- 172.21.15.60:0/491780664 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbd840762d0 0x7fbd84078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:26.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.651+0000 7fbda17fa640 1 -- 172.21.15.60:0/491780664 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6582+0+0 (secure 0 0 0) 0x7fbd8c0b8320 con 0x7fbda4160600 2024-05-09T05:01:26.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.651+0000 7fbda8b83640 1 --2- 172.21.15.60:0/491780664 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbd840762d0 0x7fbd84078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:26.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.652+0000 7fbda8b83640 1 --2- 172.21.15.60:0/491780664 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbd840762d0 0x7fbd84078790 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c006fd0 tx=0x7fbd9c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:26.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.656+0000 7fbda17fa640 1 -- 172.21.15.60:0/491780664 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbd8c081890 con 0x7fbda4160600 2024-05-09T05:01:26.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:26.849+0000 7fbdaade7640 1 -- 172.21.15.60:0/491780664 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbda40d2c10 con 0x7fbda4160600 2024-05-09T05:01:27.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.803+0000 7fbda17fa640 1 -- 172.21.15.60:0/491780664 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v41) ==== 181+0+0 (secure 0 0 0) 0x7fbd8c081190 con 0x7fbda4160600 2024-05-09T05:01:27.804 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2024-05-09T05:01:27.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.806+0000 7fbdaade7640 1 -- 172.21.15.60:0/491780664 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbd840762d0 msgr2=0x7fbd84078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:27.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.806+0000 7fbdaade7640 1 --2- 172.21.15.60:0/491780664 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbd840762d0 0x7fbd84078790 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c006fd0 tx=0x7fbd9c008040 comp rx=0 tx=0).stop 2024-05-09T05:01:27.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.806+0000 7fbdaade7640 1 -- 172.21.15.60:0/491780664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbda4160600 msgr2=0x7fbda415eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:27.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.806+0000 7fbdaade7640 1 --2- 172.21.15.60:0/491780664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbda4160600 0x7fbda415eae0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0090b0 tx=0x7fbd8c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:01:27.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.806+0000 7fbdaade7640 1 -- 172.21.15.60:0/491780664 shutdown_connections 2024-05-09T05:01:27.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.806+0000 7fbdaade7640 1 --2- 172.21.15.60:0/491780664 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbd840762d0 0x7fbd84078790 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:27.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.806+0000 7fbdaade7640 1 --2- 172.21.15.60:0/491780664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbda4160600 0x7fbda415eae0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:27.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.806+0000 7fbdaade7640 1 --2- 172.21.15.60:0/491780664 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbda40b8f40 0x7fbda4145510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:27.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.806+0000 7fbdaade7640 1 --2- 172.21.15.60:0/491780664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbda40b45a0 0x7fbda40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:27.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.806+0000 7fbdaade7640 1 -- 172.21.15.60:0/491780664 >> 172.21.15.60:0/491780664 conn(0x7fbda40b3c80 msgr2=0x7fbda415d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:27.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.806+0000 7fbdaade7640 1 -- 172.21.15.60:0/491780664 shutdown_connections 2024-05-09T05:01:27.807 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.806+0000 7fbdaade7640 1 -- 172.21.15.60:0/491780664 wait complete. 2024-05-09T05:01:27.821 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-09T05:01:27.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.935+0000 7f4a79a64640 1 Processor -- start 2024-05-09T05:01:27.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.935+0000 7f4a79a64640 1 -- start start 2024-05-09T05:01:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.936+0000 7f4a79a64640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a7415f0f0 0x7f4a740b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.936+0000 7f4a79a64640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4a740b54b0 0x7f4a740b8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.936+0000 7f4a79a64640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a740b5990 con 0x7f4a7415f4d0 2024-05-09T05:01:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.936+0000 7f4a79a64640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f4a740b65a0 con 0x7f4a740b54b0 2024-05-09T05:01:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.936+0000 7f4a79a64640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4a740b5c70 con 0x7f4a7415f0f0 2024-05-09T05:01:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.936+0000 7f4a72ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4a740b54b0 0x7f4a740b8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.936+0000 7f4a72ffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4a740b54b0 0x7f4a740b8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59800/0 (socket says 172.21.15.60:59800) 2024-05-09T05:01:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.936+0000 7f4a72ffd640 1 -- 172.21.15.60:0/1893751502 learned_addr learned my addr 172.21.15.60:0/1893751502 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.936+0000 7f4a737fe640 1 --2- 172.21.15.60:0/1893751502 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a7415f0f0 0x7f4a740b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.936+0000 7f4a72ffd640 1 -- 172.21.15.60:0/1893751502 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a7415f0f0 msgr2=0x7f4a740b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.936+0000 7f4a72ffd640 1 --2- 172.21.15.60:0/1893751502 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a7415f0f0 0x7f4a740b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.937+0000 7f4a72ffd640 1 -- 172.21.15.60:0/1893751502 >> v1:172.21.15.114:6789/0 conn(0x7f4a7415f4d0 legacy=0x7f4a740ba1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.937+0000 7f4a72ffd640 1 -- 172.21.15.60:0/1893751502 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a740b5f50 con 0x7f4a740b54b0 2024-05-09T05:01:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.937+0000 7f4a72ffd640 1 --2- 172.21.15.60:0/1893751502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a740b54b0 0x7f4a740b8f60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009870 tx=0x7f4a6802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=240c2d64a6c034d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.937+0000 7f4a727fc640 1 -- 172.21.15.60:0/1893751502 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a68037070 con 0x7f4a740b54b0 2024-05-09T05:01:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.937+0000 7f4a727fc640 1 -- 172.21.15.60:0/1893751502 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a68032440 con 0x7f4a740b54b0 2024-05-09T05:01:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.938+0000 7f4a727fc640 1 -- 172.21.15.60:0/1893751502 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a68032810 con 0x7f4a740b54b0 2024-05-09T05:01:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.938+0000 7f4a79a64640 1 -- 172.21.15.60:0/1893751502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a740b54b0 msgr2=0x7f4a740b8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.938+0000 7f4a79a64640 1 --2- 172.21.15.60:0/1893751502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a740b54b0 0x7f4a740b8f60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009870 tx=0x7f4a6802a270 comp rx=0 tx=0).stop 2024-05-09T05:01:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.938+0000 7f4a79a64640 1 -- 172.21.15.60:0/1893751502 shutdown_connections 2024-05-09T05:01:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.938+0000 7f4a79a64640 1 --2- 172.21.15.60:0/1893751502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a740b54b0 0x7f4a740b8f60 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.938+0000 7f4a79a64640 1 --2- 172.21.15.60:0/1893751502 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a7415f0f0 0x7f4a740b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.938+0000 7f4a79a64640 1 -- 172.21.15.60:0/1893751502 >> 172.21.15.60:0/1893751502 conn(0x7f4a7406ede0 msgr2=0x7f4a74073820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.938+0000 7f4a79a64640 1 -- 172.21.15.60:0/1893751502 shutdown_connections 2024-05-09T05:01:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.938+0000 7f4a79a64640 1 -- 172.21.15.60:0/1893751502 wait complete. 2024-05-09T05:01:27.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.939+0000 7f4a79a64640 1 Processor -- start 2024-05-09T05:01:27.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.940+0000 7f4a79a64640 1 -- start start 2024-05-09T05:01:27.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.940+0000 7f4a79a64640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a740b54b0 0x7f4a7414dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:27.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.940+0000 7f4a79a64640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a7415f0f0 0x7f4a7415cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:27.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.940+0000 7f4a79a64640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a7415f4d0 0x7f4a7415d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:27.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.940+0000 7f4a79a64640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4a740b9620 con 0x7f4a7415f0f0 2024-05-09T05:01:27.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.940+0000 7f4a79a64640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4a740b94a0 con 0x7f4a740b54b0 2024-05-09T05:01:27.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.940+0000 7f4a79a64640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4a740ba8b0 con 0x7f4a7415f4d0 2024-05-09T05:01:27.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.941+0000 7f4a72ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a7415f0f0 0x7f4a7415cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:27.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.941+0000 7f4a72ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a7415f0f0 0x7f4a7415cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59804/0 (socket says 172.21.15.60:59804) 2024-05-09T05:01:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.941+0000 7f4a72ffd640 1 -- 172.21.15.60:0/676086849 learned_addr learned my addr 172.21.15.60:0/676086849 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.941+0000 7f4a737fe640 1 --2- 172.21.15.60:0/676086849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a740b54b0 0x7f4a7414dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.941+0000 7f4a73fff640 1 --2- 172.21.15.60:0/676086849 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a7415f4d0 0x7f4a7415d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.941+0000 7f4a72ffd640 1 -- 172.21.15.60:0/676086849 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a7415f4d0 msgr2=0x7f4a7415d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.941+0000 7f4a72ffd640 1 --2- 172.21.15.60:0/676086849 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a7415f4d0 0x7f4a7415d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.941+0000 7f4a72ffd640 1 -- 172.21.15.60:0/676086849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a740b54b0 msgr2=0x7f4a7414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.941+0000 7f4a72ffd640 1 --2- 172.21.15.60:0/676086849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a740b54b0 0x7f4a7414dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.941+0000 7f4a72ffd640 1 -- 172.21.15.60:0/676086849 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a740af7a0 con 0x7f4a7415f0f0 2024-05-09T05:01:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.941+0000 7f4a737fe640 1 --2- 172.21.15.60:0/676086849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a740b54b0 0x7f4a7414dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.941+0000 7f4a73fff640 1 --2- 172.21.15.60:0/676086849 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a7415f4d0 0x7f4a7415d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.941+0000 7f4a72ffd640 1 --2- 172.21.15.60:0/676086849 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a7415f0f0 0x7f4a7415cf30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4a68000c00 tx=0x7f4a680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.941+0000 7f4a5bfff640 1 -- 172.21.15.60:0/676086849 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a68041020 con 0x7f4a7415f0f0 2024-05-09T05:01:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.941+0000 7f4a5bfff640 1 -- 172.21.15.60:0/676086849 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a680306c0 con 0x7f4a7415f0f0 2024-05-09T05:01:27.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.941+0000 7f4a79a64640 1 -- 172.21.15.60:0/676086849 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a740aff60 con 0x7f4a7415f0f0 2024-05-09T05:01:27.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.941+0000 7f4a5bfff640 1 -- 172.21.15.60:0/676086849 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a68037040 con 0x7f4a7415f0f0 2024-05-09T05:01:27.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.941+0000 7f4a79a64640 1 -- 172.21.15.60:0/676086849 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a740b0520 con 0x7f4a7415f0f0 2024-05-09T05:01:27.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.942+0000 7f4a79a64640 1 -- 172.21.15.60:0/676086849 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a74051040 con 0x7f4a7415f0f0 2024-05-09T05:01:27.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.943+0000 7f4a5bfff640 1 -- 172.21.15.60:0/676086849 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4a68044020 con 0x7f4a7415f0f0 2024-05-09T05:01:27.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.943+0000 7f4a5bfff640 1 --2- 172.21.15.60:0/676086849 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4a600762d0 0x7f4a60078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:27.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.943+0000 7f4a5bfff640 1 -- 172.21.15.60:0/676086849 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6144+0+0 (secure 0 0 0) 0x7f4a680b7fb0 con 0x7f4a7415f0f0 2024-05-09T05:01:27.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.944+0000 7f4a737fe640 1 --2- 172.21.15.60:0/676086849 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4a600762d0 0x7f4a60078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:27.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.944+0000 7f4a737fe640 1 --2- 172.21.15.60:0/676086849 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4a600762d0 0x7f4a60078790 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4a64006fd0 tx=0x7f4a64008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:27.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:27.948+0000 7f4a5bfff640 1 -- 172.21.15.60:0/676086849 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4a6804b030 con 0x7f4a7415f0f0 2024-05-09T05:01:28.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.142+0000 7f4a79a64640 1 -- 172.21.15.60:0/676086849 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4a74073820 con 0x7f4a7415f0f0 2024-05-09T05:01:28.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.808+0000 7f4a5bfff640 1 -- 172.21.15.60:0/676086849 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v42) ==== 169+0+0 (secure 0 0 0) 0x7f4a680817f0 con 0x7f4a7415f0f0 2024-05-09T05:01:28.809 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2024-05-09T05:01:28.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.812+0000 7f4a79a64640 1 -- 172.21.15.60:0/676086849 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4a600762d0 msgr2=0x7f4a60078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:28.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.812+0000 7f4a79a64640 1 --2- 172.21.15.60:0/676086849 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4a600762d0 0x7f4a60078790 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4a64006fd0 tx=0x7f4a64008040 comp rx=0 tx=0).stop 2024-05-09T05:01:28.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.812+0000 7f4a79a64640 1 -- 172.21.15.60:0/676086849 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a7415f0f0 msgr2=0x7f4a7415cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:28.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.812+0000 7f4a79a64640 1 --2- 172.21.15.60:0/676086849 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a7415f0f0 0x7f4a7415cf30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4a68000c00 tx=0x7f4a680046c0 comp rx=0 tx=0).stop 2024-05-09T05:01:28.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.812+0000 7f4a79a64640 1 -- 172.21.15.60:0/676086849 shutdown_connections 2024-05-09T05:01:28.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.812+0000 7f4a79a64640 1 --2- 172.21.15.60:0/676086849 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f4a600762d0 0x7f4a60078790 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:28.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.812+0000 7f4a79a64640 1 --2- 172.21.15.60:0/676086849 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4a7415f4d0 0x7f4a7415d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:28.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.812+0000 7f4a79a64640 1 --2- 172.21.15.60:0/676086849 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a7415f0f0 0x7f4a7415cf30 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:28.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.812+0000 7f4a79a64640 1 --2- 172.21.15.60:0/676086849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a740b54b0 0x7f4a7414dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:28.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.812+0000 7f4a79a64640 1 -- 172.21.15.60:0/676086849 >> 172.21.15.60:0/676086849 conn(0x7f4a7406ede0 msgr2=0x7f4a740d3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:28.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.813+0000 7f4a79a64640 1 -- 172.21.15.60:0/676086849 shutdown_connections 2024-05-09T05:01:28.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.813+0000 7f4a79a64640 1 -- 172.21.15.60:0/676086849 wait complete. 2024-05-09T05:01:28.828 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm data_new data_new --yes-i-really-really-mean-it 2024-05-09T05:01:28.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.942+0000 7f9ee3b40640 1 Processor -- start 2024-05-09T05:01:28.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.942+0000 7f9ee3b40640 1 -- start start 2024-05-09T05:01:28.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.943+0000 7f9ee3b40640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9edc15ce70 0x7f9edc0b4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:28.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.943+0000 7f9ee3b40640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9edc1600f0 0x7f9edc0d0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:28.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.943+0000 7f9ee3b40640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9edc15d250 con 0x7f9edc0b45a0 2024-05-09T05:01:28.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.943+0000 7f9ee3b40640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f9edc0b49e0 con 0x7f9edc1600f0 2024-05-09T05:01:28.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.943+0000 7f9ee3b40640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f9edc160730 con 0x7f9edc15ce70 2024-05-09T05:01:28.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.943+0000 7f9ee20dd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9edc1600f0 0x7f9edc0d0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:28.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.943+0000 7f9ee10db640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9edc0b45a0 0x7f9edc15f9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42262/0 (socket says 172.21.15.60:42262) 2024-05-09T05:01:28.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.943+0000 7f9ee10db640 1 -- 172.21.15.60:0/1905399551 learned_addr learned my addr 172.21.15.60:0/1905399551 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:28.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.943+0000 7f9ee20dd640 1 -- 172.21.15.60:0/1905399551 >> v2:172.21.15.114:3300/0 conn(0x7f9edc15ce70 msgr2=0x7f9edc0b4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:28.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.943+0000 7f9ee20dd640 1 --2- 172.21.15.60:0/1905399551 >> v2:172.21.15.114:3300/0 conn(0x7f9edc15ce70 0x7f9edc0b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:28.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.943+0000 7f9ee20dd640 1 -- 172.21.15.60:0/1905399551 >> v1:172.21.15.60:6789/0 conn(0x7f9edc0b45a0 legacy=0x7f9edc15f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:28.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.943+0000 7f9ee20dd640 1 -- 172.21.15.60:0/1905399551 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9edc06c370 con 0x7f9edc1600f0 2024-05-09T05:01:28.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.943+0000 7f9ee08da640 1 -- 172.21.15.60:0/1905399551 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3888333187 0 0) 0x7f9edc15d250 con 0x7f9edc0b45a0 2024-05-09T05:01:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.944+0000 7f9ee20dd640 1 --2- 172.21.15.60:0/1905399551 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9edc1600f0 0x7f9edc0d0c60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0b4b60 tx=0x7f9ecc02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aed32246939ebcb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.944+0000 7f9ee08da640 1 -- 172.21.15.60:0/1905399551 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc031600 con 0x7f9edc1600f0 2024-05-09T05:01:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.944+0000 7f9ee08da640 1 -- 172.21.15.60:0/1905399551 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ecc0317a0 con 0x7f9edc1600f0 2024-05-09T05:01:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.944+0000 7f9ee08da640 1 -- 172.21.15.60:0/1905399551 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc031b70 con 0x7f9edc1600f0 2024-05-09T05:01:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.945+0000 7f9ee3b40640 1 -- 172.21.15.60:0/1905399551 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9edc1600f0 msgr2=0x7f9edc0d0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.945+0000 7f9ee3b40640 1 --2- 172.21.15.60:0/1905399551 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9edc1600f0 0x7f9edc0d0c60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0b4b60 tx=0x7f9ecc02a5d0 comp rx=0 tx=0).stop 2024-05-09T05:01:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.945+0000 7f9ee3b40640 1 -- 172.21.15.60:0/1905399551 shutdown_connections 2024-05-09T05:01:28.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.945+0000 7f9ee3b40640 1 --2- 172.21.15.60:0/1905399551 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9edc1600f0 0x7f9edc0d0c60 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:28.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.945+0000 7f9ee3b40640 1 --2- 172.21.15.60:0/1905399551 >> v2:172.21.15.114:3300/0 conn(0x7f9edc15ce70 0x7f9edc0b4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:28.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.945+0000 7f9ee3b40640 1 -- 172.21.15.60:0/1905399551 >> 172.21.15.60:0/1905399551 conn(0x7f9edc0b3c80 msgr2=0x7f9edc070590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:28.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.945+0000 7f9ee3b40640 1 -- 172.21.15.60:0/1905399551 shutdown_connections 2024-05-09T05:01:28.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.945+0000 7f9ee3b40640 1 -- 172.21.15.60:0/1905399551 wait complete. 2024-05-09T05:01:28.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.946+0000 7f9ee3b40640 1 Processor -- start 2024-05-09T05:01:28.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.946+0000 7f9ee3b40640 1 -- start start 2024-05-09T05:01:28.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.947+0000 7f9ee3b40640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9edc0b45a0 0x7f9edc1f5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:28.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.947+0000 7f9ee3b40640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9edc15ce70 0x7f9edc150b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:28.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.947+0000 7f9ee3b40640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9edc1600f0 0x7f9edc154660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:28.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.947+0000 7f9ee3b40640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc06d130 con 0x7f9edc15ce70 2024-05-09T05:01:28.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.947+0000 7f9ee3b40640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc06c930 con 0x7f9edc1600f0 2024-05-09T05:01:28.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.947+0000 7f9ee3b40640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9edc06d2b0 con 0x7f9edc0b45a0 2024-05-09T05:01:28.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.947+0000 7f9ee10db640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9edc15ce70 0x7f9edc150b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:28.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.947+0000 7f9ee10db640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9edc15ce70 0x7f9edc150b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41806/0 (socket says 172.21.15.60:41806) 2024-05-09T05:01:28.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.947+0000 7f9ee10db640 1 -- 172.21.15.60:0/2431844854 learned_addr learned my addr 172.21.15.60:0/2431844854 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:28.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.947+0000 7f9ee10db640 1 -- 172.21.15.60:0/2431844854 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9edc0b45a0 msgr2=0x7f9edc1f5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:28.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.947+0000 7f9ee10db640 1 --2- 172.21.15.60:0/2431844854 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9edc0b45a0 0x7f9edc1f5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.947+0000 7f9ee10db640 1 -- 172.21.15.60:0/2431844854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9edc1600f0 msgr2=0x7f9edc154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.947+0000 7f9ee10db640 1 --2- 172.21.15.60:0/2431844854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9edc1600f0 0x7f9edc154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.947+0000 7f9ee10db640 1 -- 172.21.15.60:0/2431844854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9edc06e580 con 0x7f9edc15ce70 2024-05-09T05:01:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.948+0000 7f9ee10db640 1 --2- 172.21.15.60:0/2431844854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9edc15ce70 0x7f9edc150b10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9ed000c450 tx=0x7f9ed000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.948+0000 7f9eda7fc640 1 -- 172.21.15.60:0/2431844854 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0018070 con 0x7f9edc15ce70 2024-05-09T05:01:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.948+0000 7f9eda7fc640 1 -- 172.21.15.60:0/2431844854 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ed00074e0 con 0x7f9edc15ce70 2024-05-09T05:01:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.948+0000 7f9eda7fc640 1 -- 172.21.15.60:0/2431844854 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0004a20 con 0x7f9edc15ce70 2024-05-09T05:01:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.948+0000 7f9ee3b40640 1 -- 172.21.15.60:0/2431844854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9edc06e860 con 0x7f9edc15ce70 2024-05-09T05:01:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.948+0000 7f9ee3b40640 1 -- 172.21.15.60:0/2431844854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9edc06ee20 con 0x7f9edc15ce70 2024-05-09T05:01:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.949+0000 7f9ee3b40640 1 -- 172.21.15.60:0/2431844854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9edc06f250 con 0x7f9edc15ce70 2024-05-09T05:01:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.950+0000 7f9eda7fc640 1 -- 172.21.15.60:0/2431844854 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9ed001b070 con 0x7f9edc15ce70 2024-05-09T05:01:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.950+0000 7f9eda7fc640 1 --2- 172.21.15.60:0/2431844854 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9ec80762d0 0x7f9ec8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.951+0000 7f9eda7fc640 1 -- 172.21.15.60:0/2431844854 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5730+0+0 (secure 0 0 0) 0x7f9ed00968c0 con 0x7f9edc15ce70 2024-05-09T05:01:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.951+0000 7f9ee18dc640 1 --2- 172.21.15.60:0/2431844854 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9ec80762d0 0x7f9ec8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:28.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.951+0000 7f9ee18dc640 1 --2- 172.21.15.60:0/2431844854 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9ec80762d0 0x7f9ec8078790 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009710 tx=0x7f9ed4006c90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:28.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:28.955+0000 7f9eda7fc640 1 -- 172.21.15.60:0/2431844854 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9ed0060350 con 0x7f9edc15ce70 2024-05-09T05:01:29.149 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.148+0000 7f9ee3b40640 1 -- 172.21.15.60:0/2431844854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9edc06d9d0 con 0x7f9edc15ce70 2024-05-09T05:01:29.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.813+0000 7f9eda7fc640 1 -- 172.21.15.60:0/2431844854 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true}]=0 pool 'data_new' removed v43) ==== 160+0+0 (secure 0 0 0) 0x7f9ed005fc50 con 0x7f9edc15ce70 2024-05-09T05:01:29.814 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'data_new' removed 2024-05-09T05:01:29.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.818+0000 7f9ee3b40640 1 -- 172.21.15.60:0/2431844854 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9ec80762d0 msgr2=0x7f9ec8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:29.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.818+0000 7f9ee3b40640 1 --2- 172.21.15.60:0/2431844854 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9ec80762d0 0x7f9ec8078790 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009710 tx=0x7f9ed4006c90 comp rx=0 tx=0).stop 2024-05-09T05:01:29.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.818+0000 7f9ee3b40640 1 -- 172.21.15.60:0/2431844854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9edc15ce70 msgr2=0x7f9edc150b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:29.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.818+0000 7f9ee3b40640 1 --2- 172.21.15.60:0/2431844854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9edc15ce70 0x7f9edc150b10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9ed000c450 tx=0x7f9ed000c910 comp rx=0 tx=0).stop 2024-05-09T05:01:29.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.818+0000 7f9ee3b40640 1 -- 172.21.15.60:0/2431844854 shutdown_connections 2024-05-09T05:01:29.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.818+0000 7f9ee3b40640 1 --2- 172.21.15.60:0/2431844854 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9ec80762d0 0x7f9ec8078790 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:29.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.818+0000 7f9ee3b40640 1 --2- 172.21.15.60:0/2431844854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9edc1600f0 0x7f9edc154660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:29.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.818+0000 7f9ee3b40640 1 --2- 172.21.15.60:0/2431844854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9edc15ce70 0x7f9edc150b10 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:29.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.818+0000 7f9ee3b40640 1 --2- 172.21.15.60:0/2431844854 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9edc0b45a0 0x7f9edc1f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:29.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.818+0000 7f9ee3b40640 1 -- 172.21.15.60:0/2431844854 >> 172.21.15.60:0/2431844854 conn(0x7f9edc0b3c80 msgr2=0x7f9edc15d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:29.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.818+0000 7f9ee3b40640 1 -- 172.21.15.60:0/2431844854 shutdown_connections 2024-05-09T05:01:29.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.818+0000 7f9ee3b40640 1 -- 172.21.15.60:0/2431844854 wait complete. 2024-05-09T05:01:29.834 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-05-09T05:01:29.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.945+0000 7effb22d1640 1 Processor -- start 2024-05-09T05:01:29.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.945+0000 7effb22d1640 1 -- start start 2024-05-09T05:01:29.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.946+0000 7effb22d1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac070250 0x7effac0d2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:29.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.946+0000 7effb22d1640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effac06f6a0 con 0x7effac15f490 2024-05-09T05:01:29.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.946+0000 7effb22d1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effac06e8e0 con 0x7effac0d2980 2024-05-09T05:01:29.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.946+0000 7effb22d1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7effac06ebc0 con 0x7effac070250 2024-05-09T05:01:29.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.946+0000 7effabfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7effac15f490 0x7effac06fb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42268/0 (socket says 172.21.15.60:42268) 2024-05-09T05:01:29.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.946+0000 7effabfff640 1 -- 172.21.15.60:0/3400289600 learned_addr learned my addr 172.21.15.60:0/3400289600 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:29.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.946+0000 7effab7fe640 1 --2- 172.21.15.60:0/3400289600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac070250 0x7effac0d2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:29.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effaaffd640 1 -- 172.21.15.60:0/3400289600 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1179855233 0 0) 0x7effac06f6a0 con 0x7effac15f490 2024-05-09T05:01:29.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effaaffd640 1 -- 172.21.15.60:0/3400289600 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff94003520 con 0x7effac15f490 2024-05-09T05:01:29.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effaaffd640 1 -- 172.21.15.60:0/3400289600 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2592450454 0 0) 0x7effac06e8e0 con 0x7effac0d2980 2024-05-09T05:01:29.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effaaffd640 1 -- 172.21.15.60:0/3400289600 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effac06f6a0 con 0x7effac0d2980 2024-05-09T05:01:29.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effaaffd640 1 -- 172.21.15.60:0/3400289600 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3070758143 0 0) 0x7eff94003520 con 0x7effac15f490 2024-05-09T05:01:29.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effaaffd640 1 -- 172.21.15.60:0/3400289600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac070250 msgr2=0x7effac0d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:29.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effaaffd640 1 --2- 172.21.15.60:0/3400289600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac070250 0x7effac0d2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:29.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effaaffd640 1 -- 172.21.15.60:0/3400289600 >> v1:172.21.15.114:6789/0 conn(0x7effac0d2980 legacy=0x7effac072850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:29.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effaaffd640 1 -- 172.21.15.60:0/3400289600 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effac06eea0 con 0x7effac15f490 2024-05-09T05:01:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effaaffd640 1 -- 172.21.15.60:0/3400289600 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7eff9c0016f0 con 0x7effac15f490 2024-05-09T05:01:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effaaffd640 1 -- 172.21.15.60:0/3400289600 >> v1:172.21.15.60:6789/0 conn(0x7effac15f490 legacy=0x7effac06fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effaaffd640 1 --2- 172.21.15.60:0/3400289600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac070250 0x7eff94003c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effaaffd640 1 --2- 172.21.15.60:0/3400289600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff940041c0 0x7eff94004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effaaffd640 1 --2- 172.21.15.60:0/3400289600 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff94004d40 0x7eff940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effaaffd640 1 -- 172.21.15.60:0/3400289600 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7effac06ebc0 con 0x7eff94004d40 2024-05-09T05:01:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effaaffd640 1 -- 172.21.15.60:0/3400289600 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7eff9400bc20 con 0x7eff940041c0 2024-05-09T05:01:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effaaffd640 1 -- 172.21.15.60:0/3400289600 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7eff9400be20 con 0x7effac070250 2024-05-09T05:01:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effabfff640 1 --2- 172.21.15.60:0/3400289600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac070250 0x7eff94003c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effb086e640 1 --2- 172.21.15.60:0/3400289600 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff94004d40 0x7eff940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effab7fe640 1 --2- 172.21.15.60:0/3400289600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff940041c0 0x7eff94004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effb086e640 1 -- 172.21.15.60:0/3400289600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac070250 msgr2=0x7eff94003c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effb086e640 1 --2- 172.21.15.60:0/3400289600 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac070250 0x7eff94003c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effb086e640 1 -- 172.21.15.60:0/3400289600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff940041c0 msgr2=0x7eff94004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effb086e640 1 --2- 172.21.15.60:0/3400289600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff940041c0 0x7eff94004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.947+0000 7effb086e640 1 -- 172.21.15.60:0/3400289600 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff94003520 con 0x7eff94004d40 2024-05-09T05:01:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.948+0000 7effb086e640 1 --2- 172.21.15.60:0/3400289600 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff94004d40 0x7eff940051c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7effa4001fa0 tx=0x7eff94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcd07f2a855af4c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.948+0000 7effaaffd640 1 -- 172.21.15.60:0/3400289600 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa4001bc0 con 0x7eff94004d40 2024-05-09T05:01:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.948+0000 7effaaffd640 1 -- 172.21.15.60:0/3400289600 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effa4006d70 con 0x7eff94004d40 2024-05-09T05:01:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.948+0000 7effaaffd640 1 -- 172.21.15.60:0/3400289600 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa4003bb0 con 0x7eff94004d40 2024-05-09T05:01:29.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.949+0000 7effb22d1640 1 -- 172.21.15.60:0/3400289600 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff94004d40 msgr2=0x7eff940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.949+0000 7effb22d1640 1 --2- 172.21.15.60:0/3400289600 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff94004d40 0x7eff940051c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7effa4001fa0 tx=0x7eff94005830 comp rx=0 tx=0).stop 2024-05-09T05:01:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.949+0000 7effb086e640 1 -- 172.21.15.60:0/3400289600 reap_dead start 2024-05-09T05:01:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.949+0000 7effb22d1640 1 -- 172.21.15.60:0/3400289600 shutdown_connections 2024-05-09T05:01:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.949+0000 7effb22d1640 1 -- 172.21.15.60:0/3400289600 >> 172.21.15.60:0/3400289600 conn(0x7effac09b8c0 msgr2=0x7effac09bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.949+0000 7effb22d1640 1 -- 172.21.15.60:0/3400289600 shutdown_connections 2024-05-09T05:01:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.949+0000 7effb22d1640 1 -- 172.21.15.60:0/3400289600 wait complete. 2024-05-09T05:01:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.950+0000 7effb22d1640 1 Processor -- start 2024-05-09T05:01:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.950+0000 7effb22d1640 1 -- start start 2024-05-09T05:01:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.951+0000 7effb22d1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac15a3b0 0x7effac15dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.951+0000 7effb22d1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effac15a790 0x7effac15e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.951+0000 7effb22d1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effac0b2f50 0x7effac0b33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.951+0000 7effb22d1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7effac0d2440 con 0x7effac15a790 2024-05-09T05:01:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.951+0000 7effb22d1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7effac15f870 con 0x7effac0b2f50 2024-05-09T05:01:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.951+0000 7effb22d1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7effac0d25c0 con 0x7effac15a3b0 2024-05-09T05:01:29.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.951+0000 7effab7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effac15a790 0x7effac15e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:29.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.951+0000 7effab7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effac15a790 0x7effac15e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41836/0 (socket says 172.21.15.60:41836) 2024-05-09T05:01:29.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.951+0000 7effab7fe640 1 -- 172.21.15.60:0/4284290447 learned_addr learned my addr 172.21.15.60:0/4284290447 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:29.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.951+0000 7effabfff640 1 --2- 172.21.15.60:0/4284290447 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac15a3b0 0x7effac15dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:29.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.951+0000 7effb086e640 1 --2- 172.21.15.60:0/4284290447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effac0b2f50 0x7effac0b33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:29.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.951+0000 7effab7fe640 1 -- 172.21.15.60:0/4284290447 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac15a3b0 msgr2=0x7effac15dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:29.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.951+0000 7effab7fe640 1 --2- 172.21.15.60:0/4284290447 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac15a3b0 0x7effac15dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:29.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.951+0000 7effab7fe640 1 -- 172.21.15.60:0/4284290447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effac0b2f50 msgr2=0x7effac0b33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:29.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.951+0000 7effab7fe640 1 --2- 172.21.15.60:0/4284290447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effac0b2f50 0x7effac0b33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:29.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.951+0000 7effab7fe640 1 -- 172.21.15.60:0/4284290447 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effac0b1cc0 con 0x7effac15a790 2024-05-09T05:01:29.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.952+0000 7effab7fe640 1 --2- 172.21.15.60:0/4284290447 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effac15a790 0x7effac15e190 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7effa00119a0 tx=0x7effa0011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:29.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.952+0000 7effa8ff9640 1 -- 172.21.15.60:0/4284290447 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa000faf0 con 0x7effac15a790 2024-05-09T05:01:29.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.952+0000 7effa8ff9640 1 -- 172.21.15.60:0/4284290447 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effa000fc90 con 0x7effac15a790 2024-05-09T05:01:29.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.952+0000 7effb22d1640 1 -- 172.21.15.60:0/4284290447 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effac0b1fa0 con 0x7effac15a790 2024-05-09T05:01:29.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.952+0000 7effa8ff9640 1 -- 172.21.15.60:0/4284290447 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa0014440 con 0x7effac15a790 2024-05-09T05:01:29.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.952+0000 7effb22d1640 1 -- 172.21.15.60:0/4284290447 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effac0b2560 con 0x7effac15a790 2024-05-09T05:01:29.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.953+0000 7effb22d1640 1 -- 172.21.15.60:0/4284290447 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effac051040 con 0x7effac15a790 2024-05-09T05:01:29.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.953+0000 7effa8ff9640 1 -- 172.21.15.60:0/4284290447 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7effa0012040 con 0x7effac15a790 2024-05-09T05:01:29.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.954+0000 7effa8ff9640 1 --2- 172.21.15.60:0/4284290447 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7eff980762d0 0x7eff98078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:29.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.954+0000 7effa8ff9640 1 -- 172.21.15.60:0/4284290447 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5335+0+0 (secure 0 0 0) 0x7effa0097b30 con 0x7effac15a790 2024-05-09T05:01:29.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.954+0000 7effabfff640 1 --2- 172.21.15.60:0/4284290447 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7eff980762d0 0x7eff98078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:29.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.954+0000 7effabfff640 1 --2- 172.21.15.60:0/4284290447 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7eff980762d0 0x7eff98078790 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7effac071430 tx=0x7eff9c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:29.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:29.958+0000 7effa8ff9640 1 -- 172.21.15.60:0/4284290447 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7effa0019030 con 0x7effac15a790 2024-05-09T05:01:30.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.149+0000 7effb22d1640 1 -- 172.21.15.60:0/4284290447 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7effac099070 con 0x7effac15a790 2024-05-09T05:01:30.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.152+0000 7effa8ff9640 1 -- 172.21.15.60:0/4284290447 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) ==== 104+0+0 (secure 0 0 0) 0x7effa0061600 con 0x7effac15a790 2024-05-09T05:01:30.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.155+0000 7effb22d1640 1 -- 172.21.15.60:0/4284290447 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7eff980762d0 msgr2=0x7eff98078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:30.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.155+0000 7effb22d1640 1 --2- 172.21.15.60:0/4284290447 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7eff980762d0 0x7eff98078790 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7effac071430 tx=0x7eff9c009040 comp rx=0 tx=0).stop 2024-05-09T05:01:30.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.155+0000 7effb22d1640 1 -- 172.21.15.60:0/4284290447 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effac15a790 msgr2=0x7effac15e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:30.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.155+0000 7effb22d1640 1 --2- 172.21.15.60:0/4284290447 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effac15a790 0x7effac15e190 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7effa00119a0 tx=0x7effa0011e60 comp rx=0 tx=0).stop 2024-05-09T05:01:30.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.155+0000 7effb22d1640 1 -- 172.21.15.60:0/4284290447 shutdown_connections 2024-05-09T05:01:30.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.155+0000 7effb22d1640 1 --2- 172.21.15.60:0/4284290447 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7eff980762d0 0x7eff98078790 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:30.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.155+0000 7effb22d1640 1 --2- 172.21.15.60:0/4284290447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effac0b2f50 0x7effac0b33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:30.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.155+0000 7effb22d1640 1 --2- 172.21.15.60:0/4284290447 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effac15a790 0x7effac15e190 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:30.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.155+0000 7effb22d1640 1 --2- 172.21.15.60:0/4284290447 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7effac15a3b0 0x7effac15dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:30.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.155+0000 7effb22d1640 1 -- 172.21.15.60:0/4284290447 >> 172.21.15.60:0/4284290447 conn(0x7effac09b8c0 msgr2=0x7effac15ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:30.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.155+0000 7effb22d1640 1 -- 172.21.15.60:0/4284290447 shutdown_connections 2024-05-09T05:01:30.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.155+0000 7effb22d1640 1 -- 172.21.15.60:0/4284290447 wait complete. 2024-05-09T05:01:30.167 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_symlink_recovery 2024-05-09T05:01:30.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.272+0000 7fac07171640 1 Processor -- start 2024-05-09T05:01:30.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.272+0000 7fac07171640 1 -- start start 2024-05-09T05:01:30.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.273+0000 7fac07171640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fac0015f3c0 0x7fac0009ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:30.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.273+0000 7fac07171640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fac000b4d30 0x7fac00072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:30.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.273+0000 7fac07171640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac0015f7a0 con 0x7fac00073fb0 2024-05-09T05:01:30.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.273+0000 7fac07171640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fac00071cb0 con 0x7fac0015f3c0 2024-05-09T05:01:30.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.273+0000 7fac07171640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fac0009f480 con 0x7fac000b4d30 2024-05-09T05:01:30.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.273+0000 7fac04f0d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fac0015f3c0 0x7fac0009ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:30.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.273+0000 7fac04f0d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fac0015f3c0 0x7fac0009ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41848/0 (socket says 172.21.15.60:41848) 2024-05-09T05:01:30.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.273+0000 7fac04f0d640 1 -- 172.21.15.60:0/190394764 learned_addr learned my addr 172.21.15.60:0/190394764 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:30.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.273+0000 7fac0570e640 1 --2- 172.21.15.60:0/190394764 >> v2:172.21.15.114:3300/0 conn(0x7fac000b4d30 0x7fac00072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:30.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.274+0000 7fac04f0d640 1 -- 172.21.15.60:0/190394764 >> v2:172.21.15.114:3300/0 conn(0x7fac000b4d30 msgr2=0x7fac00072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:30.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.274+0000 7fac04f0d640 1 --2- 172.21.15.60:0/190394764 >> v2:172.21.15.114:3300/0 conn(0x7fac000b4d30 0x7fac00072140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:30.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.274+0000 7fac04f0d640 1 -- 172.21.15.60:0/190394764 >> v1:172.21.15.114:6789/0 conn(0x7fac00073fb0 legacy=0x7fac000b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:30.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.274+0000 7fac04f0d640 1 -- 172.21.15.60:0/190394764 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac0009f760 con 0x7fac0015f3c0 2024-05-09T05:01:30.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.274+0000 7fac0570e640 1 --2- 172.21.15.60:0/190394764 >> v2:172.21.15.114:3300/0 conn(0x7fac000b4d30 0x7fac00072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:30.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.274+0000 7fac04f0d640 1 --2- 172.21.15.60:0/190394764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac0015f3c0 0x7fac0009ebe0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fac000d52d0 tx=0x7fabf002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eeeca7d62a68b39 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:30.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.274+0000 7fabff7fe640 1 -- 172.21.15.60:0/190394764 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf0037070 con 0x7fac0015f3c0 2024-05-09T05:01:30.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.275+0000 7fabff7fe640 1 -- 172.21.15.60:0/190394764 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabf0032440 con 0x7fac0015f3c0 2024-05-09T05:01:30.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.275+0000 7fabff7fe640 1 -- 172.21.15.60:0/190394764 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf0032810 con 0x7fac0015f3c0 2024-05-09T05:01:30.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.275+0000 7fac07171640 1 -- 172.21.15.60:0/190394764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac0015f3c0 msgr2=0x7fac0009ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:30.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.275+0000 7fac07171640 1 --2- 172.21.15.60:0/190394764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac0015f3c0 0x7fac0009ebe0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fac000d52d0 tx=0x7fabf002a140 comp rx=0 tx=0).stop 2024-05-09T05:01:30.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.275+0000 7fac07171640 1 -- 172.21.15.60:0/190394764 shutdown_connections 2024-05-09T05:01:30.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.275+0000 7fac07171640 1 --2- 172.21.15.60:0/190394764 >> v2:172.21.15.114:3300/0 conn(0x7fac000b4d30 0x7fac00072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:30.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.275+0000 7fac07171640 1 --2- 172.21.15.60:0/190394764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac0015f3c0 0x7fac0009ebe0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:30.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.275+0000 7fac07171640 1 -- 172.21.15.60:0/190394764 >> 172.21.15.60:0/190394764 conn(0x7fac000d1470 msgr2=0x7fac000d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:30.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.275+0000 7fac07171640 1 -- 172.21.15.60:0/190394764 shutdown_connections 2024-05-09T05:01:30.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.275+0000 7fac07171640 1 -- 172.21.15.60:0/190394764 wait complete. 2024-05-09T05:01:30.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.276+0000 7fac07171640 1 Processor -- start 2024-05-09T05:01:30.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.276+0000 7fac07171640 1 -- start start 2024-05-09T05:01:30.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.277+0000 7fac07171640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac00073fb0 0x7fac000b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:30.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.277+0000 7fac07171640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac000b4d30 0x7fac0015b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:30.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.277+0000 7fac07171640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac0015f3c0 0x7fac0015c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:30.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.277+0000 7fac07171640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fac00072680 con 0x7fac0015f3c0 2024-05-09T05:01:30.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.277+0000 7fac07171640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fac0009f120 con 0x7fac000b4d30 2024-05-09T05:01:30.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.277+0000 7fac07171640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fac00072800 con 0x7fac00073fb0 2024-05-09T05:01:30.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.277+0000 7fac0570e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac0015f3c0 0x7fac0015c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:30.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.277+0000 7fac0570e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac0015f3c0 0x7fac0015c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41864/0 (socket says 172.21.15.60:41864) 2024-05-09T05:01:30.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.277+0000 7fac0570e640 1 -- 172.21.15.60:0/469097851 learned_addr learned my addr 172.21.15.60:0/469097851 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:30.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.277+0000 7fac04f0d640 1 --2- 172.21.15.60:0/469097851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac00073fb0 0x7fac000b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:30.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.277+0000 7fabfffff640 1 --2- 172.21.15.60:0/469097851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac000b4d30 0x7fac0015b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:30.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.278+0000 7fac0570e640 1 -- 172.21.15.60:0/469097851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac00073fb0 msgr2=0x7fac000b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:30.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.278+0000 7fac0570e640 1 --2- 172.21.15.60:0/469097851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac00073fb0 0x7fac000b1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:30.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.278+0000 7fac0570e640 1 -- 172.21.15.60:0/469097851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac000b4d30 msgr2=0x7fac0015b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:30.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.278+0000 7fac0570e640 1 --2- 172.21.15.60:0/469097851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac000b4d30 0x7fac0015b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:30.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.278+0000 7fac0570e640 1 -- 172.21.15.60:0/469097851 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac0015b910 con 0x7fac0015f3c0 2024-05-09T05:01:30.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.278+0000 7fac04f0d640 1 --2- 172.21.15.60:0/469097851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac00073fb0 0x7fac000b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:30.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.278+0000 7fac0570e640 1 --2- 172.21.15.60:0/469097851 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac0015f3c0 0x7fac0015c680 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fabf8009fd0 tx=0x7fabf800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:30.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.278+0000 7fabfd7fa640 1 -- 172.21.15.60:0/469097851 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf801a070 con 0x7fac0015f3c0 2024-05-09T05:01:30.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.278+0000 7fabfd7fa640 1 -- 172.21.15.60:0/469097851 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabf8003bb0 con 0x7fac0015f3c0 2024-05-09T05:01:30.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.278+0000 7fac07171640 1 -- 172.21.15.60:0/469097851 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac001f50c0 con 0x7fac0015f3c0 2024-05-09T05:01:30.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.278+0000 7fabfd7fa640 1 -- 172.21.15.60:0/469097851 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf8004030 con 0x7fac0015f3c0 2024-05-09T05:01:30.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.278+0000 7fac07171640 1 -- 172.21.15.60:0/469097851 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac001f5680 con 0x7fac0015f3c0 2024-05-09T05:01:30.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.279+0000 7fac07171640 1 -- 172.21.15.60:0/469097851 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac00051040 con 0x7fac0015f3c0 2024-05-09T05:01:30.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.280+0000 7fabfd7fa640 1 -- 172.21.15.60:0/469097851 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fabf801d070 con 0x7fac0015f3c0 2024-05-09T05:01:30.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.280+0000 7fabfd7fa640 1 --2- 172.21.15.60:0/469097851 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fabe00762d0 0x7fabe0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:30.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.280+0000 7fabfd7fa640 1 -- 172.21.15.60:0/469097851 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5335+0+0 (secure 0 0 0) 0x7fabf8097780 con 0x7fac0015f3c0 2024-05-09T05:01:30.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.280+0000 7fac04f0d640 1 --2- 172.21.15.60:0/469097851 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fabe00762d0 0x7fabe0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:30.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.281+0000 7fac04f0d640 1 --2- 172.21.15.60:0/469097851 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fabe00762d0 0x7fabe0078790 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fabf0034000 tx=0x7fabf002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:30.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.284+0000 7fabfd7fa640 1 -- 172.21.15.60:0/469097851 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fabf8010070 con 0x7fac0015f3c0 2024-05-09T05:01:30.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.475+0000 7fac07171640 1 -- 172.21.15.60:0/469097851 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_symlink_recovery"} v 0) -- 0x7fac0009db10 con 0x7fac0015f3c0 2024-05-09T05:01:30.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.479+0000 7fabfd7fa640 1 -- 172.21.15.60:0/469097851 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_symlink_recovery"}]=0 v14) ==== 103+0+0 (secure 0 0 0) 0x7fabf8061250 con 0x7fac0015f3c0 2024-05-09T05:01:30.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.481+0000 7fac07171640 1 -- 172.21.15.60:0/469097851 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fabe00762d0 msgr2=0x7fabe0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:30.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.481+0000 7fac07171640 1 --2- 172.21.15.60:0/469097851 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fabe00762d0 0x7fabe0078790 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fabf0034000 tx=0x7fabf002e000 comp rx=0 tx=0).stop 2024-05-09T05:01:30.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.481+0000 7fac07171640 1 -- 172.21.15.60:0/469097851 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac0015f3c0 msgr2=0x7fac0015c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:30.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.481+0000 7fac07171640 1 --2- 172.21.15.60:0/469097851 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac0015f3c0 0x7fac0015c680 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fabf8009fd0 tx=0x7fabf800d560 comp rx=0 tx=0).stop 2024-05-09T05:01:30.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.481+0000 7fac07171640 1 -- 172.21.15.60:0/469097851 shutdown_connections 2024-05-09T05:01:30.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.481+0000 7fac07171640 1 --2- 172.21.15.60:0/469097851 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fabe00762d0 0x7fabe0078790 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:30.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.481+0000 7fac07171640 1 --2- 172.21.15.60:0/469097851 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac0015f3c0 0x7fac0015c680 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:30.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.481+0000 7fac07171640 1 --2- 172.21.15.60:0/469097851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac000b4d30 0x7fac0015b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:30.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.481+0000 7fac07171640 1 --2- 172.21.15.60:0/469097851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fac00073fb0 0x7fac000b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:30.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.481+0000 7fac07171640 1 -- 172.21.15.60:0/469097851 >> 172.21.15.60:0/469097851 conn(0x7fac000d1470 msgr2=0x7fac0009ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:30.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.481+0000 7fac07171640 1 -- 172.21.15.60:0/469097851 shutdown_connections 2024-05-09T05:01:30.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.481+0000 7fac07171640 1 -- 172.21.15.60:0/469097851 wait complete. 2024-05-09T05:01:30.493 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_backtrace.TestBacktrace.test_backtrace' 2024-05-09T05:01:30.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.596+0000 7f9dcddf0640 1 Processor -- start 2024-05-09T05:01:30.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.596+0000 7f9dcddf0640 1 -- start start 2024-05-09T05:01:30.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.597+0000 7f9dcddf0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9dc815f5b0 0x7f9dc80b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:30.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.597+0000 7f9dcddf0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dc80b2600 con 0x7f9dc8098640 2024-05-09T05:01:30.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.597+0000 7f9dcddf0640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dc80b28e0 con 0x7f9dc80b16b0 2024-05-09T05:01:30.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.597+0000 7f9dcddf0640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f9dc80b23d0 con 0x7f9dc815f5b0 2024-05-09T05:01:30.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.597+0000 7f9dc77fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9dc815f5b0 0x7f9dc80b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:30.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.597+0000 7f9dc6ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9dc8098640 0x7f9dc809b280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42280/0 (socket says 172.21.15.60:42280) 2024-05-09T05:01:30.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.597+0000 7f9dc6ffd640 1 -- 172.21.15.60:0/872545908 learned_addr learned my addr 172.21.15.60:0/872545908 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:30.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.597+0000 7f9dc77fe640 1 -- 172.21.15.60:0/872545908 >> v1:172.21.15.114:6789/0 conn(0x7f9dc80b16b0 legacy=0x7f9dc809b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:30.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.597+0000 7f9dc77fe640 1 -- 172.21.15.60:0/872545908 >> v1:172.21.15.60:6789/0 conn(0x7f9dc8098640 legacy=0x7f9dc809b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:30.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.597+0000 7f9dc67fc640 1 -- 172.21.15.60:0/872545908 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2750326160 0 0) 0x7f9dc80b2600 con 0x7f9dc8098640 2024-05-09T05:01:30.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.597+0000 7f9dc77fe640 1 -- 172.21.15.60:0/872545908 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc80b2bc0 con 0x7f9dc815f5b0 2024-05-09T05:01:30.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.598+0000 7f9dc77fe640 1 --2- 172.21.15.60:0/872545908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc815f5b0 0x7f9dc80b1e90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9db80097a0 tx=0x7f9db802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7c1287835c2558 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:30.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.598+0000 7f9dc67fc640 1 -- 172.21.15.60:0/872545908 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db8037070 con 0x7f9dc815f5b0 2024-05-09T05:01:30.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.598+0000 7f9dc67fc640 1 -- 172.21.15.60:0/872545908 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9db8032440 con 0x7f9dc815f5b0 2024-05-09T05:01:30.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.598+0000 7f9dc67fc640 1 -- 172.21.15.60:0/872545908 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db8032810 con 0x7f9dc815f5b0 2024-05-09T05:01:30.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.599+0000 7f9dcddf0640 1 -- 172.21.15.60:0/872545908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc815f5b0 msgr2=0x7f9dc80b1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:30.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.599+0000 7f9dcddf0640 1 --2- 172.21.15.60:0/872545908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc815f5b0 0x7f9dc80b1e90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9db80097a0 tx=0x7f9db802a210 comp rx=0 tx=0).stop 2024-05-09T05:01:30.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.599+0000 7f9dcddf0640 1 -- 172.21.15.60:0/872545908 shutdown_connections 2024-05-09T05:01:30.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.599+0000 7f9dcddf0640 1 --2- 172.21.15.60:0/872545908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc815f5b0 0x7f9dc80b1e90 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:30.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.599+0000 7f9dcddf0640 1 -- 172.21.15.60:0/872545908 >> 172.21.15.60:0/872545908 conn(0x7f9dc80b42e0 msgr2=0x7f9dc80b46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:30.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.599+0000 7f9dcddf0640 1 -- 172.21.15.60:0/872545908 shutdown_connections 2024-05-09T05:01:30.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.599+0000 7f9dcddf0640 1 -- 172.21.15.60:0/872545908 wait complete. 2024-05-09T05:01:30.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.600+0000 7f9dcddf0640 1 Processor -- start 2024-05-09T05:01:30.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.600+0000 7f9dcddf0640 1 -- start start 2024-05-09T05:01:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.601+0000 7f9dcddf0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dc8098640 0x7f9dc81511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.601+0000 7f9dcddf0640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dc80b16b0 0x7f9dc81f4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:30.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.601+0000 7f9dcddf0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc815f5b0 0x7f9dc815b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.601+0000 7f9dcddf0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc80b3180 con 0x7f9dc815f5b0 2024-05-09T05:01:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.601+0000 7f9dcddf0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc80b36a0 con 0x7f9dc8098640 2024-05-09T05:01:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.601+0000 7f9dcddf0640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9dc80b3820 con 0x7f9dc80b16b0 2024-05-09T05:01:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.601+0000 7f9dc7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc815f5b0 0x7f9dc815b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.601+0000 7f9dc7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc815f5b0 0x7f9dc815b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41882/0 (socket says 172.21.15.60:41882) 2024-05-09T05:01:30.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.601+0000 7f9dc7fff640 1 -- 172.21.15.60:0/1835923054 learned_addr learned my addr 172.21.15.60:0/1835923054 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:30.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.601+0000 7f9dc6ffd640 1 --2- 172.21.15.60:0/1835923054 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dc80b16b0 0x7f9dc81f4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:30.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.601+0000 7f9dc77fe640 1 --2- 172.21.15.60:0/1835923054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dc8098640 0x7f9dc81511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:30.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.601+0000 7f9dc7fff640 1 -- 172.21.15.60:0/1835923054 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dc80b16b0 msgr2=0x7f9dc81f4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:30.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.601+0000 7f9dc7fff640 1 --2- 172.21.15.60:0/1835923054 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dc80b16b0 0x7f9dc81f4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:30.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.601+0000 7f9dc7fff640 1 -- 172.21.15.60:0/1835923054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dc8098640 msgr2=0x7f9dc81511c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:30.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.601+0000 7f9dc7fff640 1 --2- 172.21.15.60:0/1835923054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dc8098640 0x7f9dc81511c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:30.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.601+0000 7f9dc7fff640 1 -- 172.21.15.60:0/1835923054 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc806c830 con 0x7f9dc815f5b0 2024-05-09T05:01:30.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.602+0000 7f9dc7fff640 1 --2- 172.21.15.60:0/1835923054 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc815f5b0 0x7f9dc815b1e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f9dc000c7b0 tx=0x7f9dc000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:30.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.602+0000 7f9daffff640 1 -- 172.21.15.60:0/1835923054 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0004b60 con 0x7f9dc815f5b0 2024-05-09T05:01:30.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.602+0000 7f9daffff640 1 -- 172.21.15.60:0/1835923054 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dc0004d00 con 0x7f9dc815f5b0 2024-05-09T05:01:30.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.602+0000 7f9dcddf0640 1 -- 172.21.15.60:0/1835923054 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dc80ab820 con 0x7f9dc815f5b0 2024-05-09T05:01:30.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.602+0000 7f9daffff640 1 -- 172.21.15.60:0/1835923054 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc000aa90 con 0x7f9dc815f5b0 2024-05-09T05:01:30.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.602+0000 7f9dcddf0640 1 -- 172.21.15.60:0/1835923054 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dc81f4c70 con 0x7f9dc815f5b0 2024-05-09T05:01:30.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.603+0000 7f9dcddf0640 1 -- 172.21.15.60:0/1835923054 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dc8051040 con 0x7f9dc815f5b0 2024-05-09T05:01:30.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.603+0000 7f9daffff640 1 -- 172.21.15.60:0/1835923054 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9dc000ac30 con 0x7f9dc815f5b0 2024-05-09T05:01:30.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.604+0000 7f9daffff640 1 --2- 172.21.15.60:0/1835923054 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9db40762d0 0x7f9db4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:30.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.604+0000 7f9daffff640 1 -- 172.21.15.60:0/1835923054 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5335+0+0 (secure 0 0 0) 0x7f9dc0097ab0 con 0x7f9dc815f5b0 2024-05-09T05:01:30.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.604+0000 7f9dc77fe640 1 --2- 172.21.15.60:0/1835923054 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9db40762d0 0x7f9db4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:30.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.605+0000 7f9dc77fe640 1 --2- 172.21.15.60:0/1835923054 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9db40762d0 0x7f9db4078790 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9db8034000 tx=0x7f9db802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:30.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.608+0000 7f9daffff640 1 -- 172.21.15.60:0/1835923054 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9dc00614e0 con 0x7f9dc815f5b0 2024-05-09T05:01:30.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:30.874+0000 7f9dcddf0640 1 -- 172.21.15.60:0/1835923054 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_backtrace.TestBacktrace.test_backtrace"]} v 0) -- 0x7f9dc80b3da0 con 0x7f9dc815f5b0 2024-05-09T05:01:31.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.819+0000 7f9daffff640 1 -- 172.21.15.60:0/1835923054 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_backtrace.TestBacktrace.test_backtrace"]}]=0 v88) ==== 135+0+0 (secure 0 0 0) 0x7f9dc0060de0 con 0x7f9dc815f5b0 2024-05-09T05:01:31.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.822+0000 7f9dcddf0640 1 -- 172.21.15.60:0/1835923054 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9db40762d0 msgr2=0x7f9db4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:31.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.822+0000 7f9dcddf0640 1 --2- 172.21.15.60:0/1835923054 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9db40762d0 0x7f9db4078790 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9db8034000 tx=0x7f9db802e000 comp rx=0 tx=0).stop 2024-05-09T05:01:31.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.822+0000 7f9dcddf0640 1 -- 172.21.15.60:0/1835923054 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc815f5b0 msgr2=0x7f9dc815b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:31.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.822+0000 7f9dcddf0640 1 --2- 172.21.15.60:0/1835923054 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc815f5b0 0x7f9dc815b1e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f9dc000c7b0 tx=0x7f9dc000cc70 comp rx=0 tx=0).stop 2024-05-09T05:01:31.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.822+0000 7f9dcddf0640 1 -- 172.21.15.60:0/1835923054 shutdown_connections 2024-05-09T05:01:31.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.822+0000 7f9dcddf0640 1 --2- 172.21.15.60:0/1835923054 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9db40762d0 0x7f9db4078790 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:31.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.822+0000 7f9dcddf0640 1 --2- 172.21.15.60:0/1835923054 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dc815f5b0 0x7f9dc815b1e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:31.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.822+0000 7f9dcddf0640 1 --2- 172.21.15.60:0/1835923054 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9dc80b16b0 0x7f9dc81f4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:31.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.822+0000 7f9dcddf0640 1 --2- 172.21.15.60:0/1835923054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dc8098640 0x7f9dc81511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:31.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.822+0000 7f9dcddf0640 1 -- 172.21.15.60:0/1835923054 >> 172.21.15.60:0/1835923054 conn(0x7f9dc80b42e0 msgr2=0x7f9dc815d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:31.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.822+0000 7f9dcddf0640 1 -- 172.21.15.60:0/1835923054 shutdown_connections 2024-05-09T05:01:31.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.822+0000 7f9dcddf0640 1 -- 172.21.15.60:0/1835923054 wait complete. 2024-05-09T05:01:31.837 INFO:tasks.cephfs_test_runner:That the 'parent' 'layout' and 'symlink' xattrs on the head objects of files ... ok 2024-05-09T05:01:31.837 INFO:tasks.cephfs_test_runner:Starting test: test_backtrace_flush_on_deleted_data_pool (tasks.cephfs.test_backtrace.TestBacktrace) that the MDS does not go read-only when handling backtrace update errors 2024-05-09T05:01:31.837 INFO:tasks.cephfs_test_runner:test_backtrace_flush_on_deleted_data_pool (tasks.cephfs.test_backtrace.TestBacktrace) 2024-05-09T05:01:31.837 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:01:31.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.947+0000 7fefacd38640 1 Processor -- start 2024-05-09T05:01:31.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.947+0000 7fefacd38640 1 -- start start 2024-05-09T05:01:31.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.948+0000 7fefacd38640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fefa80ac430 0x7fefa8073330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:31.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.948+0000 7fefacd38640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefa8071b80 con 0x7fefa815c3a0 2024-05-09T05:01:31.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.948+0000 7fefacd38640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefa8071e60 con 0x7fefa8070860 2024-05-09T05:01:31.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.948+0000 7fefacd38640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fefa8072140 con 0x7fefa80ac430 2024-05-09T05:01:31.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.948+0000 7fefa6d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fefa80ac430 0x7fefa8073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:31.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.948+0000 7fefa659c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fefa815c3a0 0x7fefa8070150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42294/0 (socket says 172.21.15.60:42294) 2024-05-09T05:01:31.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.948+0000 7fefa659c640 1 -- 172.21.15.60:0/2018509947 learned_addr learned my addr 172.21.15.60:0/2018509947 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:31.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.948+0000 7fefa6d9d640 1 -- 172.21.15.60:0/2018509947 >> v1:172.21.15.114:6789/0 conn(0x7fefa8070860 legacy=0x7fefa8072c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:31.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.948+0000 7fefa6d9d640 1 -- 172.21.15.60:0/2018509947 >> v1:172.21.15.60:6789/0 conn(0x7fefa815c3a0 legacy=0x7fefa8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:31.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.948+0000 7fefa6d9d640 1 -- 172.21.15.60:0/2018509947 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa8072420 con 0x7fefa80ac430 2024-05-09T05:01:31.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.949+0000 7fefa6d9d640 1 --2- 172.21.15.60:0/2018509947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa80ac430 0x7fefa8073330 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fefa0009d90 tx=0x7fefa002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b51763db50e5602 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:31.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.949+0000 7fefa5d9b640 1 -- 172.21.15.60:0/2018509947 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa0007740 con 0x7fefa80ac430 2024-05-09T05:01:31.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.949+0000 7fefa5d9b640 1 -- 172.21.15.60:0/2018509947 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefa00078e0 con 0x7fefa80ac430 2024-05-09T05:01:31.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.949+0000 7fefa5d9b640 1 -- 172.21.15.60:0/2018509947 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa0007cb0 con 0x7fefa80ac430 2024-05-09T05:01:31.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.949+0000 7fefacd38640 1 -- 172.21.15.60:0/2018509947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa80ac430 msgr2=0x7fefa8073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:31.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.949+0000 7fefacd38640 1 --2- 172.21.15.60:0/2018509947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa80ac430 0x7fefa8073330 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fefa0009d90 tx=0x7fefa002a660 comp rx=0 tx=0).stop 2024-05-09T05:01:31.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.949+0000 7fefacd38640 1 -- 172.21.15.60:0/2018509947 shutdown_connections 2024-05-09T05:01:31.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.949+0000 7fefacd38640 1 --2- 172.21.15.60:0/2018509947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa80ac430 0x7fefa8073330 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:31.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.949+0000 7fefacd38640 1 -- 172.21.15.60:0/2018509947 >> 172.21.15.60:0/2018509947 conn(0x7fefa80972f0 msgr2=0x7fefa80976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:31.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.950+0000 7fefacd38640 1 -- 172.21.15.60:0/2018509947 shutdown_connections 2024-05-09T05:01:31.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.950+0000 7fefacd38640 1 -- 172.21.15.60:0/2018509947 wait complete. 2024-05-09T05:01:31.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.951+0000 7fefacd38640 1 Processor -- start 2024-05-09T05:01:31.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.951+0000 7fefacd38640 1 -- start start 2024-05-09T05:01:31.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.952+0000 7fefacd38640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fefa8070860 0x7fefa815e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:31.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.952+0000 7fefacd38640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefa80ac430 0x7fefa8096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:31.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.952+0000 7fefacd38640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa815c3a0 0x7fefa80b0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:31.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.952+0000 7fefacd38640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fefa8071810 con 0x7fefa815c3a0 2024-05-09T05:01:31.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.952+0000 7fefacd38640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fefa815c780 con 0x7fefa80ac430 2024-05-09T05:01:31.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.952+0000 7fefacd38640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fefa8097de0 con 0x7fefa8070860 2024-05-09T05:01:31.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.952+0000 7fefa6d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa815c3a0 0x7fefa80b0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:31.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.952+0000 7fefa6d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa815c3a0 0x7fefa80b0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41898/0 (socket says 172.21.15.60:41898) 2024-05-09T05:01:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.952+0000 7fefa6d9d640 1 -- 172.21.15.60:0/2841512804 learned_addr learned my addr 172.21.15.60:0/2841512804 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.952+0000 7fefa659c640 1 --2- 172.21.15.60:0/2841512804 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fefa8070860 0x7fefa815e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.952+0000 7fef9dd9b640 1 --2- 172.21.15.60:0/2841512804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefa80ac430 0x7fefa8096c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.952+0000 7fefa6d9d640 1 -- 172.21.15.60:0/2841512804 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fefa8070860 msgr2=0x7fefa815e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.952+0000 7fefa6d9d640 1 --2- 172.21.15.60:0/2841512804 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fefa8070860 0x7fefa815e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.952+0000 7fefa6d9d640 1 -- 172.21.15.60:0/2841512804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefa80ac430 msgr2=0x7fefa8096c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.952+0000 7fefa6d9d640 1 --2- 172.21.15.60:0/2841512804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefa80ac430 0x7fefa8096c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.952+0000 7fefa6d9d640 1 -- 172.21.15.60:0/2841512804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa81f4e90 con 0x7fefa815c3a0 2024-05-09T05:01:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.952+0000 7fefa659c640 1 --2- 172.21.15.60:0/2841512804 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fefa8070860 0x7fefa815e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.953+0000 7fefa6d9d640 1 --2- 172.21.15.60:0/2841512804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa815c3a0 0x7fefa80b0ec0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fefa0009d90 tx=0x7fefa0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.953+0000 7fef9f7fe640 1 -- 172.21.15.60:0/2841512804 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa0040070 con 0x7fefa815c3a0 2024-05-09T05:01:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.953+0000 7fef9f7fe640 1 -- 172.21.15.60:0/2841512804 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefa0031bc0 con 0x7fefa815c3a0 2024-05-09T05:01:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.953+0000 7fefacd38640 1 -- 172.21.15.60:0/2841512804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefa80adae0 con 0x7fefa815c3a0 2024-05-09T05:01:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.953+0000 7fef9f7fe640 1 -- 172.21.15.60:0/2841512804 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa003c420 con 0x7fefa815c3a0 2024-05-09T05:01:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.953+0000 7fefacd38640 1 -- 172.21.15.60:0/2841512804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefa80ae0a0 con 0x7fefa815c3a0 2024-05-09T05:01:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.954+0000 7fefacd38640 1 -- 172.21.15.60:0/2841512804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefa8051040 con 0x7fefa815c3a0 2024-05-09T05:01:31.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.954+0000 7fef9f7fe640 1 -- 172.21.15.60:0/2841512804 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fefa0039040 con 0x7fefa815c3a0 2024-05-09T05:01:31.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.955+0000 7fef9f7fe640 1 --2- 172.21.15.60:0/2841512804 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fef880762d0 0x7fef88078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:31.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.955+0000 7fef9f7fe640 1 -- 172.21.15.60:0/2841512804 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5335+0+0 (secure 0 0 0) 0x7fefa00b8a20 con 0x7fefa815c3a0 2024-05-09T05:01:31.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.955+0000 7fefa659c640 1 --2- 172.21.15.60:0/2841512804 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fef880762d0 0x7fef88078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:31.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.956+0000 7fefa659c640 1 --2- 172.21.15.60:0/2841512804 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fef880762d0 0x7fef88078790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fef98000f60 tx=0x7fef98009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:31.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:31.959+0000 7fef9f7fe640 1 -- 172.21.15.60:0/2841512804 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fefa00825b0 con 0x7fefa815c3a0 2024-05-09T05:01:32.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.152+0000 7fefacd38640 1 -- 172.21.15.60:0/2841512804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fefa8069510 con 0x7fefa815c3a0 2024-05-09T05:01:32.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.153+0000 7fef9f7fe640 1 -- 172.21.15.60:0/2841512804 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+13267 (secure 0 0 0) 0x7fefa0081eb0 con 0x7fefa815c3a0 2024-05-09T05:01:32.154 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:32.154 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":43,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:01:29.810058+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:6834/399160914":"2024-05-10T05:01:25.471249+0000","172.21.15.60:6832/399160914":"2024-05-10T05:01:25.471249+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:01:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.155+0000 7fefacd38640 1 -- 172.21.15.60:0/2841512804 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fef880762d0 msgr2=0x7fef88078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.155+0000 7fefacd38640 1 --2- 172.21.15.60:0/2841512804 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fef880762d0 0x7fef88078790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fef98000f60 tx=0x7fef98009040 comp rx=0 tx=0).stop 2024-05-09T05:01:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.155+0000 7fefacd38640 1 -- 172.21.15.60:0/2841512804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa815c3a0 msgr2=0x7fefa80b0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.155+0000 7fefacd38640 1 --2- 172.21.15.60:0/2841512804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa815c3a0 0x7fefa80b0ec0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fefa0009d90 tx=0x7fefa0004a40 comp rx=0 tx=0).stop 2024-05-09T05:01:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.155+0000 7fefacd38640 1 -- 172.21.15.60:0/2841512804 shutdown_connections 2024-05-09T05:01:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.155+0000 7fefacd38640 1 --2- 172.21.15.60:0/2841512804 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fef880762d0 0x7fef88078790 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.155+0000 7fefacd38640 1 --2- 172.21.15.60:0/2841512804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa815c3a0 0x7fefa80b0ec0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.155+0000 7fefacd38640 1 --2- 172.21.15.60:0/2841512804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefa80ac430 0x7fefa8096c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.155+0000 7fefacd38640 1 --2- 172.21.15.60:0/2841512804 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fefa8070860 0x7fefa815e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.155+0000 7fefacd38640 1 -- 172.21.15.60:0/2841512804 >> 172.21.15.60:0/2841512804 conn(0x7fefa80972f0 msgr2=0x7fefa80976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.155+0000 7fefacd38640 1 -- 172.21.15.60:0/2841512804 shutdown_connections 2024-05-09T05:01:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.155+0000 7fefacd38640 1 -- 172.21.15.60:0/2841512804 wait complete. 2024-05-09T05:01:32.167 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T05:00:06.925911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T05:01:32.167 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T05:01:32.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.273+0000 7ff23bbaf640 1 Processor -- start 2024-05-09T05:01:32.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.273+0000 7ff23bbaf640 1 -- start start 2024-05-09T05:01:32.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.274+0000 7ff23bbaf640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7ff23415b750 0x7ff2340cda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:32.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.274+0000 7ff23bbaf640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff234072430 0x7ff2340cf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:32.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.274+0000 7ff23bbaf640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff2340cf7c0 0x7ff2340736b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:32.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.274+0000 7ff23bbaf640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7ff23415bb30 con 0x7ff2340cf7c0 2024-05-09T05:01:32.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.274+0000 7ff23bbaf640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7ff234072bd0 con 0x7ff23415b750 2024-05-09T05:01:32.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.274+0000 7ff23bbaf640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff23406f170 con 0x7ff234072430 2024-05-09T05:01:32.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.274+0000 7ff23a14c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff2340cf7c0 0x7ff2340736b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:32.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.274+0000 7ff23a14c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff2340cf7c0 0x7ff2340736b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41902/0 (socket says 172.21.15.60:41902) 2024-05-09T05:01:32.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.274+0000 7ff23a14c640 1 -- 172.21.15.60:0/3485149671 learned_addr learned my addr 172.21.15.60:0/3485149671 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:32.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.274+0000 7ff23994b640 1 --2- 172.21.15.60:0/3485149671 >> v2:172.21.15.114:3300/0 conn(0x7ff23415b750 0x7ff2340cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:32.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.274+0000 7ff23914a640 1 --2- 172.21.15.60:0/3485149671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff234072430 0x7ff2340cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:32.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.274+0000 7ff23a14c640 1 -- 172.21.15.60:0/3485149671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff234072430 msgr2=0x7ff2340cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:32.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.274+0000 7ff23a14c640 1 --2- 172.21.15.60:0/3485149671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff234072430 0x7ff2340cf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.274+0000 7ff23a14c640 1 -- 172.21.15.60:0/3485149671 >> v2:172.21.15.114:3300/0 conn(0x7ff23415b750 msgr2=0x7ff2340cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:32.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.274+0000 7ff23a14c640 1 --2- 172.21.15.60:0/3485149671 >> v2:172.21.15.114:3300/0 conn(0x7ff23415b750 0x7ff2340cda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.274+0000 7ff23a14c640 1 -- 172.21.15.60:0/3485149671 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff23406f410 con 0x7ff2340cf7c0 2024-05-09T05:01:32.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.275+0000 7ff23a14c640 1 --2- 172.21.15.60:0/3485149671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2340cf7c0 0x7ff2340736b0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff2340b2e00 tx=0x7ff22c02c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48bbd7c625b80270 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:32.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.275+0000 7ff238949640 1 -- 172.21.15.60:0/3485149671 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c039070 con 0x7ff2340cf7c0 2024-05-09T05:01:32.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.275+0000 7ff238949640 1 -- 172.21.15.60:0/3485149671 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff22c009600 con 0x7ff2340cf7c0 2024-05-09T05:01:32.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.275+0000 7ff238949640 1 -- 172.21.15.60:0/3485149671 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c0099f0 con 0x7ff2340cf7c0 2024-05-09T05:01:32.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.276+0000 7ff23bbaf640 1 -- 172.21.15.60:0/3485149671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2340cf7c0 msgr2=0x7ff2340736b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:32.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.276+0000 7ff23bbaf640 1 --2- 172.21.15.60:0/3485149671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2340cf7c0 0x7ff2340736b0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff2340b2e00 tx=0x7ff22c02c810 comp rx=0 tx=0).stop 2024-05-09T05:01:32.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.276+0000 7ff23bbaf640 1 -- 172.21.15.60:0/3485149671 shutdown_connections 2024-05-09T05:01:32.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.276+0000 7ff23bbaf640 1 --2- 172.21.15.60:0/3485149671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2340cf7c0 0x7ff2340736b0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.276+0000 7ff23bbaf640 1 --2- 172.21.15.60:0/3485149671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff234072430 0x7ff2340cf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.276+0000 7ff23bbaf640 1 --2- 172.21.15.60:0/3485149671 >> v2:172.21.15.114:3300/0 conn(0x7ff23415b750 0x7ff2340cda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.276+0000 7ff23bbaf640 1 -- 172.21.15.60:0/3485149671 >> 172.21.15.60:0/3485149671 conn(0x7ff2340b0da0 msgr2=0x7ff2340b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:32.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.276+0000 7ff23bbaf640 1 -- 172.21.15.60:0/3485149671 shutdown_connections 2024-05-09T05:01:32.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.276+0000 7ff23bbaf640 1 -- 172.21.15.60:0/3485149671 wait complete. 2024-05-09T05:01:32.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.277+0000 7ff23bbaf640 1 Processor -- start 2024-05-09T05:01:32.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.277+0000 7ff23bbaf640 1 -- start start 2024-05-09T05:01:32.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.278+0000 7ff23bbaf640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff234072430 0x7ff2341f57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:32.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.278+0000 7ff23bbaf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2340cf7c0 0x7ff23406fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:32.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.278+0000 7ff23bbaf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff23415b750 0x7ff234070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:32.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.278+0000 7ff23bbaf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff234072890 con 0x7ff2340cf7c0 2024-05-09T05:01:32.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.278+0000 7ff23bbaf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff23406f9d0 con 0x7ff23415b750 2024-05-09T05:01:32.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.278+0000 7ff23bbaf640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff234073bf0 con 0x7ff234072430 2024-05-09T05:01:32.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.278+0000 7ff23914a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2340cf7c0 0x7ff23406fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:32.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.278+0000 7ff23914a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2340cf7c0 0x7ff23406fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41912/0 (socket says 172.21.15.60:41912) 2024-05-09T05:01:32.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.278+0000 7ff23914a640 1 -- 172.21.15.60:0/1020902007 learned_addr learned my addr 172.21.15.60:0/1020902007 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:32.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.278+0000 7ff23994b640 1 --2- 172.21.15.60:0/1020902007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff234072430 0x7ff2341f57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:32.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.278+0000 7ff23a14c640 1 --2- 172.21.15.60:0/1020902007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff23415b750 0x7ff234070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:32.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.278+0000 7ff23914a640 1 -- 172.21.15.60:0/1020902007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff234072430 msgr2=0x7ff2341f57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:32.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.278+0000 7ff23914a640 1 --2- 172.21.15.60:0/1020902007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff234072430 0x7ff2341f57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.278+0000 7ff23914a640 1 -- 172.21.15.60:0/1020902007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff23415b750 msgr2=0x7ff234070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:32.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.278+0000 7ff23914a640 1 --2- 172.21.15.60:0/1020902007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff23415b750 0x7ff234070430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.278+0000 7ff23914a640 1 -- 172.21.15.60:0/1020902007 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2341f5d30 con 0x7ff2340cf7c0 2024-05-09T05:01:32.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.279+0000 7ff23914a640 1 --2- 172.21.15.60:0/1020902007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2340cf7c0 0x7ff23406fef0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff22400e9b0 tx=0x7ff22400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:32.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.279+0000 7ff22a7fc640 1 -- 172.21.15.60:0/1020902007 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff224017070 con 0x7ff2340cf7c0 2024-05-09T05:01:32.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.279+0000 7ff22a7fc640 1 -- 172.21.15.60:0/1020902007 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff22400c9a0 con 0x7ff2340cf7c0 2024-05-09T05:01:32.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.279+0000 7ff22a7fc640 1 -- 172.21.15.60:0/1020902007 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22400cd90 con 0x7ff2340cf7c0 2024-05-09T05:01:32.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.279+0000 7ff23bbaf640 1 -- 172.21.15.60:0/1020902007 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2341e2ea0 con 0x7ff2340cf7c0 2024-05-09T05:01:32.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.279+0000 7ff23bbaf640 1 -- 172.21.15.60:0/1020902007 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2341e3460 con 0x7ff2340cf7c0 2024-05-09T05:01:32.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.280+0000 7ff23bbaf640 1 -- 172.21.15.60:0/1020902007 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff234051040 con 0x7ff2340cf7c0 2024-05-09T05:01:32.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.281+0000 7ff22a7fc640 1 -- 172.21.15.60:0/1020902007 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff22401a070 con 0x7ff2340cf7c0 2024-05-09T05:01:32.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.281+0000 7ff22a7fc640 1 --2- 172.21.15.60:0/1020902007 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff2180762d0 0x7ff218078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:32.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.281+0000 7ff22a7fc640 1 -- 172.21.15.60:0/1020902007 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5335+0+0 (secure 0 0 0) 0x7ff224097390 con 0x7ff2340cf7c0 2024-05-09T05:01:32.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.281+0000 7ff23994b640 1 --2- 172.21.15.60:0/1020902007 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff2180762d0 0x7ff218078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:32.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.282+0000 7ff23994b640 1 --2- 172.21.15.60:0/1020902007 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff2180762d0 0x7ff218078790 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff230009a20 tx=0x7ff230009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:32.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.286+0000 7ff22a7fc640 1 -- 172.21.15.60:0/1020902007 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff2341f5d30 con 0x7ff2340cf7c0 2024-05-09T05:01:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.478+0000 7ff23bbaf640 1 -- 172.21.15.60:0/1020902007 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff234147870 con 0x7ff2340cf7c0 2024-05-09T05:01:32.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.478+0000 7ff22a7fc640 1 -- 172.21.15.60:0/1020902007 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) ==== 93+0+10 (secure 0 0 0) 0x7ff2240608a0 con 0x7ff2340cf7c0 2024-05-09T05:01:32.479 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-05-09T05:01:32.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.480+0000 7ff23bbaf640 1 -- 172.21.15.60:0/1020902007 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff2180762d0 msgr2=0x7ff218078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:32.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.480+0000 7ff23bbaf640 1 --2- 172.21.15.60:0/1020902007 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff2180762d0 0x7ff218078790 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff230009a20 tx=0x7ff230009190 comp rx=0 tx=0).stop 2024-05-09T05:01:32.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.480+0000 7ff23bbaf640 1 -- 172.21.15.60:0/1020902007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2340cf7c0 msgr2=0x7ff23406fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:32.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.480+0000 7ff23bbaf640 1 --2- 172.21.15.60:0/1020902007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2340cf7c0 0x7ff23406fef0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff22400e9b0 tx=0x7ff22400ee70 comp rx=0 tx=0).stop 2024-05-09T05:01:32.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.480+0000 7ff23bbaf640 1 -- 172.21.15.60:0/1020902007 shutdown_connections 2024-05-09T05:01:32.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.480+0000 7ff23bbaf640 1 --2- 172.21.15.60:0/1020902007 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff2180762d0 0x7ff218078790 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.480+0000 7ff23bbaf640 1 --2- 172.21.15.60:0/1020902007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff23415b750 0x7ff234070430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.480+0000 7ff23bbaf640 1 --2- 172.21.15.60:0/1020902007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2340cf7c0 0x7ff23406fef0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.480+0000 7ff23bbaf640 1 --2- 172.21.15.60:0/1020902007 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff234072430 0x7ff2341f57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.480+0000 7ff23bbaf640 1 -- 172.21.15.60:0/1020902007 >> 172.21.15.60:0/1020902007 conn(0x7ff2340b0da0 msgr2=0x7ff2340b6f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:32.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.480+0000 7ff23bbaf640 1 -- 172.21.15.60:0/1020902007 shutdown_connections 2024-05-09T05:01:32.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.480+0000 7ff23bbaf640 1 -- 172.21.15.60:0/1020902007 wait complete. 2024-05-09T05:01:32.492 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_backtrace.TestBacktrace.test_backtrace_flush_on_deleted_data_pool' 2024-05-09T05:01:32.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.595+0000 7f6b99590640 1 Processor -- start 2024-05-09T05:01:32.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.596+0000 7f6b99590640 1 -- start start 2024-05-09T05:01:32.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.596+0000 7f6b99590640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6b9415ad00 0x7f6b940b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:32.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.596+0000 7f6b99590640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6b9415d850 0x7f6b9406e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.596+0000 7f6b99590640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b9415df30 con 0x7f6b9406eac0 2024-05-09T05:01:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.596+0000 7f6b99590640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f6b94072e80 con 0x7f6b9415d850 2024-05-09T05:01:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.596+0000 7f6b99590640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f6b9415e7d0 con 0x7f6b9415ad00 2024-05-09T05:01:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.596+0000 7f6b927fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6b9415d850 0x7f6b9406e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.597+0000 7f6b937fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6b9406eac0 0x7f6b9409c9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42296/0 (socket says 172.21.15.60:42296) 2024-05-09T05:01:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.597+0000 7f6b937fe640 1 -- 172.21.15.60:0/562690363 learned_addr learned my addr 172.21.15.60:0/562690363 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.597+0000 7f6b92ffd640 1 --2- 172.21.15.60:0/562690363 >> v2:172.21.15.114:3300/0 conn(0x7f6b9415ad00 0x7f6b940b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.597+0000 7f6b927fc640 1 -- 172.21.15.60:0/562690363 >> v2:172.21.15.114:3300/0 conn(0x7f6b9415ad00 msgr2=0x7f6b940b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.597+0000 7f6b927fc640 1 --2- 172.21.15.60:0/562690363 >> v2:172.21.15.114:3300/0 conn(0x7f6b9415ad00 0x7f6b940b38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.597+0000 7f6b927fc640 1 -- 172.21.15.60:0/562690363 >> v1:172.21.15.60:6789/0 conn(0x7f6b9406eac0 legacy=0x7f6b9409c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.597+0000 7f6b927fc640 1 -- 172.21.15.60:0/562690363 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9415e210 con 0x7f6b9415d850 2024-05-09T05:01:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.597+0000 7f6b91ffb640 1 -- 172.21.15.60:0/562690363 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2799810380 0 0) 0x7f6b9415df30 con 0x7f6b9406eac0 2024-05-09T05:01:32.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.597+0000 7f6b92ffd640 1 --2- 172.21.15.60:0/562690363 >> v2:172.21.15.114:3300/0 conn(0x7f6b9415ad00 0x7f6b940b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:32.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.598+0000 7f6b927fc640 1 --2- 172.21.15.60:0/562690363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b9415d850 0x7f6b9406e580 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6b94075e10 tx=0x7f6b7c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f5e854f0f90d15 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:32.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.598+0000 7f6b91ffb640 1 -- 172.21.15.60:0/562690363 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c037070 con 0x7f6b9415d850 2024-05-09T05:01:32.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.598+0000 7f6b91ffb640 1 -- 172.21.15.60:0/562690363 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b7c032440 con 0x7f6b9415d850 2024-05-09T05:01:32.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.598+0000 7f6b91ffb640 1 -- 172.21.15.60:0/562690363 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c032810 con 0x7f6b9415d850 2024-05-09T05:01:32.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.598+0000 7f6b99590640 1 -- 172.21.15.60:0/562690363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b9415d850 msgr2=0x7f6b9406e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:32.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.598+0000 7f6b99590640 1 --2- 172.21.15.60:0/562690363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b9415d850 0x7f6b9406e580 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6b94075e10 tx=0x7f6b7c02a1a0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.598+0000 7f6b99590640 1 -- 172.21.15.60:0/562690363 shutdown_connections 2024-05-09T05:01:32.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.598+0000 7f6b99590640 1 --2- 172.21.15.60:0/562690363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b9415d850 0x7f6b9406e580 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.598+0000 7f6b99590640 1 --2- 172.21.15.60:0/562690363 >> v2:172.21.15.114:3300/0 conn(0x7f6b9415ad00 0x7f6b940b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.598+0000 7f6b99590640 1 -- 172.21.15.60:0/562690363 >> 172.21.15.60:0/562690363 conn(0x7f6b94071e80 msgr2=0x7f6b94072290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:32.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.598+0000 7f6b99590640 1 -- 172.21.15.60:0/562690363 shutdown_connections 2024-05-09T05:01:32.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.598+0000 7f6b99590640 1 -- 172.21.15.60:0/562690363 wait complete. 2024-05-09T05:01:32.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.600+0000 7f6b99590640 1 Processor -- start 2024-05-09T05:01:32.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.600+0000 7f6b99590640 1 -- start start 2024-05-09T05:01:32.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.600+0000 7f6b99590640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b9406eac0 0x7f6b940c4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:32.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.600+0000 7f6b99590640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b9415ad00 0x7f6b940aded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:32.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.600+0000 7f6b99590640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b9415d850 0x7f6b940d1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:32.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.600+0000 7f6b99590640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9406eec0 con 0x7f6b9415d850 2024-05-09T05:01:32.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.600+0000 7f6b99590640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9415dce0 con 0x7f6b9415ad00 2024-05-09T05:01:32.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.600+0000 7f6b99590640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6b94160fe0 con 0x7f6b9406eac0 2024-05-09T05:01:32.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.600+0000 7f6b937fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b9415d850 0x7f6b940d1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:32.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.600+0000 7f6b937fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b9415d850 0x7f6b940d1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41926/0 (socket says 172.21.15.60:41926) 2024-05-09T05:01:32.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.600+0000 7f6b937fe640 1 -- 172.21.15.60:0/3166683647 learned_addr learned my addr 172.21.15.60:0/3166683647 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:32.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.601+0000 7f6b92ffd640 1 --2- 172.21.15.60:0/3166683647 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b9406eac0 0x7f6b940c4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:32.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.601+0000 7f6b927fc640 1 --2- 172.21.15.60:0/3166683647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b9415ad00 0x7f6b940aded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:32.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.601+0000 7f6b937fe640 1 -- 172.21.15.60:0/3166683647 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b9406eac0 msgr2=0x7f6b940c4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:32.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.601+0000 7f6b937fe640 1 --2- 172.21.15.60:0/3166683647 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b9406eac0 0x7f6b940c4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.601+0000 7f6b937fe640 1 -- 172.21.15.60:0/3166683647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b9415ad00 msgr2=0x7f6b940aded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:32.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.601+0000 7f6b937fe640 1 --2- 172.21.15.60:0/3166683647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b9415ad00 0x7f6b940aded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:32.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.601+0000 7f6b937fe640 1 -- 172.21.15.60:0/3166683647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b940c4e70 con 0x7f6b9415d850 2024-05-09T05:01:32.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.601+0000 7f6b927fc640 1 --2- 172.21.15.60:0/3166683647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b9415ad00 0x7f6b940aded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:32.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.601+0000 7f6b937fe640 1 --2- 172.21.15.60:0/3166683647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b9415d850 0x7f6b940d1030 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6b8400c440 tx=0x7f6b8400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:32.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.601+0000 7f6b8b7fe640 1 -- 172.21.15.60:0/3166683647 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b84018070 con 0x7f6b9415d850 2024-05-09T05:01:32.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.601+0000 7f6b8b7fe640 1 -- 172.21.15.60:0/3166683647 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b840074e0 con 0x7f6b9415d850 2024-05-09T05:01:32.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.601+0000 7f6b99590640 1 -- 172.21.15.60:0/3166683647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b940c5140 con 0x7f6b9415d850 2024-05-09T05:01:32.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.601+0000 7f6b8b7fe640 1 -- 172.21.15.60:0/3166683647 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b84004a20 con 0x7f6b9415d850 2024-05-09T05:01:32.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.601+0000 7f6b99590640 1 -- 172.21.15.60:0/3166683647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b940c5700 con 0x7f6b9415d850 2024-05-09T05:01:32.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.602+0000 7f6b99590640 1 -- 172.21.15.60:0/3166683647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b94051040 con 0x7f6b9415d850 2024-05-09T05:01:32.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.603+0000 7f6b8b7fe640 1 -- 172.21.15.60:0/3166683647 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6b8401b070 con 0x7f6b9415d850 2024-05-09T05:01:32.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.603+0000 7f6b8b7fe640 1 --2- 172.21.15.60:0/3166683647 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6b740762d0 0x7f6b74078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:32.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.603+0000 7f6b8b7fe640 1 -- 172.21.15.60:0/3166683647 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5335+0+0 (secure 0 0 0) 0x7f6b840965d0 con 0x7f6b9415d850 2024-05-09T05:01:32.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.604+0000 7f6b92ffd640 1 --2- 172.21.15.60:0/3166683647 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6b740762d0 0x7f6b74078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:32.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.604+0000 7f6b92ffd640 1 --2- 172.21.15.60:0/3166683647 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6b740762d0 0x7f6b74078790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6b940b2d00 tx=0x7f6b8c003a10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:32.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.608+0000 7f6b8b7fe640 1 -- 172.21.15.60:0/3166683647 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6b84060130 con 0x7f6b9415d850 2024-05-09T05:01:32.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:32.874+0000 7f6b99590640 1 -- 172.21.15.60:0/3166683647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_backtrace.TestBacktrace.test_backtrace_flush_on_deleted_data_pool"]} v 0) -- 0x7f6b940ae410 con 0x7f6b9415d850 2024-05-09T05:01:33.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.824+0000 7f6b8b7fe640 1 -- 172.21.15.60:0/3166683647 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_backtrace.TestBacktrace.test_backtrace_flush_on_deleted_data_pool"]}]=0 v90) ==== 165+0+0 (secure 0 0 0) 0x7f6b8405fa30 con 0x7f6b9415d850 2024-05-09T05:01:33.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.827+0000 7f6b99590640 1 -- 172.21.15.60:0/3166683647 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6b740762d0 msgr2=0x7f6b74078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:33.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.827+0000 7f6b99590640 1 --2- 172.21.15.60:0/3166683647 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6b740762d0 0x7f6b74078790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6b940b2d00 tx=0x7f6b8c003a10 comp rx=0 tx=0).stop 2024-05-09T05:01:33.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.827+0000 7f6b99590640 1 -- 172.21.15.60:0/3166683647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b9415d850 msgr2=0x7f6b940d1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:33.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.827+0000 7f6b99590640 1 --2- 172.21.15.60:0/3166683647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b9415d850 0x7f6b940d1030 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6b8400c440 tx=0x7f6b8400c900 comp rx=0 tx=0).stop 2024-05-09T05:01:33.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.827+0000 7f6b99590640 1 -- 172.21.15.60:0/3166683647 shutdown_connections 2024-05-09T05:01:33.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.827+0000 7f6b99590640 1 --2- 172.21.15.60:0/3166683647 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6b740762d0 0x7f6b74078790 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:33.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.827+0000 7f6b99590640 1 --2- 172.21.15.60:0/3166683647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b9415d850 0x7f6b940d1030 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:33.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.827+0000 7f6b99590640 1 --2- 172.21.15.60:0/3166683647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b9415ad00 0x7f6b940aded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:33.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.827+0000 7f6b99590640 1 --2- 172.21.15.60:0/3166683647 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6b9406eac0 0x7f6b940c4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:33.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.827+0000 7f6b99590640 1 -- 172.21.15.60:0/3166683647 >> 172.21.15.60:0/3166683647 conn(0x7f6b94071e80 msgr2=0x7f6b9415a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:33.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.828+0000 7f6b99590640 1 -- 172.21.15.60:0/3166683647 shutdown_connections 2024-05-09T05:01:33.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.828+0000 7f6b99590640 1 -- 172.21.15.60:0/3166683647 wait complete. 2024-05-09T05:01:33.842 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-09T05:01:33.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.955+0000 7f8c607a4640 1 Processor -- start 2024-05-09T05:01:33.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.955+0000 7f8c607a4640 1 -- start start 2024-05-09T05:01:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.956+0000 7f8c607a4640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8c580723a0 0x7f8c580b7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.956+0000 7f8c607a4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c5815d910 0x7f8c5815dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.956+0000 7f8c607a4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c5815e2c0 con 0x7f8c58072780 2024-05-09T05:01:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.956+0000 7f8c607a4640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f8c580c75b0 con 0x7f8c580723a0 2024-05-09T05:01:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.956+0000 7f8c607a4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8c580c7890 con 0x7f8c5815d910 2024-05-09T05:01:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.956+0000 7f8c5e540640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8c580723a0 0x7f8c580b7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.956+0000 7f8c5e540640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8c580723a0 0x7f8c580b7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41934/0 (socket says 172.21.15.60:41934) 2024-05-09T05:01:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.956+0000 7f8c5e540640 1 -- 172.21.15.60:0/1816839223 learned_addr learned my addr 172.21.15.60:0/1816839223 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.956+0000 7f8c5ed41640 1 --2- 172.21.15.60:0/1816839223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c5815d910 0x7f8c5815dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.956+0000 7f8c5e540640 1 -- 172.21.15.60:0/1816839223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c5815d910 msgr2=0x7f8c5815dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.956+0000 7f8c5e540640 1 --2- 172.21.15.60:0/1816839223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c5815d910 0x7f8c5815dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.956+0000 7f8c5e540640 1 -- 172.21.15.60:0/1816839223 >> v1:172.21.15.114:6789/0 conn(0x7f8c58072780 legacy=0x7f8c580c8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.956+0000 7f8c5e540640 1 -- 172.21.15.60:0/1816839223 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c580c7b70 con 0x7f8c580723a0 2024-05-09T05:01:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.957+0000 7f8c5e540640 1 --2- 172.21.15.60:0/1816839223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c580723a0 0x7f8c580b7430 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8c54009870 tx=0x7f8c5402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d4285573f7713ee server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:33.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.957+0000 7f8c5d53e640 1 -- 172.21.15.60:0/1816839223 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c54037070 con 0x7f8c580723a0 2024-05-09T05:01:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.957+0000 7f8c5d53e640 1 -- 172.21.15.60:0/1816839223 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c54032440 con 0x7f8c580723a0 2024-05-09T05:01:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.957+0000 7f8c5d53e640 1 -- 172.21.15.60:0/1816839223 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c54032810 con 0x7f8c580723a0 2024-05-09T05:01:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.958+0000 7f8c607a4640 1 -- 172.21.15.60:0/1816839223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c580723a0 msgr2=0x7f8c580b7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.958+0000 7f8c607a4640 1 --2- 172.21.15.60:0/1816839223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c580723a0 0x7f8c580b7430 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8c54009870 tx=0x7f8c5402a270 comp rx=0 tx=0).stop 2024-05-09T05:01:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.958+0000 7f8c607a4640 1 -- 172.21.15.60:0/1816839223 shutdown_connections 2024-05-09T05:01:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.958+0000 7f8c607a4640 1 --2- 172.21.15.60:0/1816839223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c5815d910 0x7f8c5815dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.958+0000 7f8c607a4640 1 --2- 172.21.15.60:0/1816839223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c580723a0 0x7f8c580b7430 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.958+0000 7f8c607a4640 1 -- 172.21.15.60:0/1816839223 >> 172.21.15.60:0/1816839223 conn(0x7f8c58070820 msgr2=0x7f8c580b54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.958+0000 7f8c607a4640 1 -- 172.21.15.60:0/1816839223 shutdown_connections 2024-05-09T05:01:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.958+0000 7f8c607a4640 1 -- 172.21.15.60:0/1816839223 wait complete. 2024-05-09T05:01:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.959+0000 7f8c607a4640 1 Processor -- start 2024-05-09T05:01:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.959+0000 7f8c607a4640 1 -- start start 2024-05-09T05:01:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.960+0000 7f8c607a4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c580723a0 0x7f8c580ae7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.960+0000 7f8c607a4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c58072780 0x7f8c580ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.960+0000 7f8c607a4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c5815d910 0x7f8c580cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.960+0000 7f8c607a4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8c580c7350 con 0x7f8c580723a0 2024-05-09T05:01:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.960+0000 7f8c607a4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8c580c71d0 con 0x7f8c58072780 2024-05-09T05:01:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.960+0000 7f8c607a4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8c580c8d60 con 0x7f8c5815d910 2024-05-09T05:01:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.960+0000 7f8c5e540640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c580723a0 0x7f8c580ae7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.960+0000 7f8c5e540640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c580723a0 0x7f8c580ae7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41942/0 (socket says 172.21.15.60:41942) 2024-05-09T05:01:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.960+0000 7f8c5e540640 1 -- 172.21.15.60:0/4113366641 learned_addr learned my addr 172.21.15.60:0/4113366641 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.960+0000 7f8c5e540640 1 -- 172.21.15.60:0/4113366641 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c5815d910 msgr2=0x7f8c580cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.960+0000 7f8c5e540640 1 --2- 172.21.15.60:0/4113366641 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c5815d910 0x7f8c580cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.960+0000 7f8c5dd3f640 1 --2- 172.21.15.60:0/4113366641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c58072780 0x7f8c580ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.960+0000 7f8c5e540640 1 -- 172.21.15.60:0/4113366641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c58072780 msgr2=0x7f8c580ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.960+0000 7f8c5e540640 1 --2- 172.21.15.60:0/4113366641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c58072780 0x7f8c580ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.960+0000 7f8c5e540640 1 -- 172.21.15.60:0/4113366641 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c580aa600 con 0x7f8c580723a0 2024-05-09T05:01:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.960+0000 7f8c5dd3f640 1 --2- 172.21.15.60:0/4113366641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c58072780 0x7f8c580ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.961+0000 7f8c5e540640 1 --2- 172.21.15.60:0/4113366641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c580723a0 0x7f8c580ae7d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8c54000c00 tx=0x7f8c540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.961+0000 7f8c4effd640 1 -- 172.21.15.60:0/4113366641 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c54041020 con 0x7f8c580723a0 2024-05-09T05:01:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.961+0000 7f8c4effd640 1 -- 172.21.15.60:0/4113366641 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c540306c0 con 0x7f8c580723a0 2024-05-09T05:01:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.961+0000 7f8c607a4640 1 -- 172.21.15.60:0/4113366641 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c580aa8c0 con 0x7f8c580723a0 2024-05-09T05:01:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.961+0000 7f8c4effd640 1 -- 172.21.15.60:0/4113366641 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c54037040 con 0x7f8c580723a0 2024-05-09T05:01:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.961+0000 7f8c607a4640 1 -- 172.21.15.60:0/4113366641 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c580ab3a0 con 0x7f8c580723a0 2024-05-09T05:01:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.962+0000 7f8c607a4640 1 -- 172.21.15.60:0/4113366641 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c58051040 con 0x7f8c580723a0 2024-05-09T05:01:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.962+0000 7f8c4effd640 1 -- 172.21.15.60:0/4113366641 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8c54044020 con 0x7f8c580723a0 2024-05-09T05:01:33.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.963+0000 7f8c4effd640 1 --2- 172.21.15.60:0/4113366641 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8c3c0762d0 0x7f8c3c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:33.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.963+0000 7f8c4effd640 1 -- 172.21.15.60:0/4113366641 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5335+0+0 (secure 0 0 0) 0x7f8c540b7c40 con 0x7f8c580723a0 2024-05-09T05:01:33.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.963+0000 7f8c5dd3f640 1 --2- 172.21.15.60:0/4113366641 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8c3c0762d0 0x7f8c3c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:33.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.963+0000 7f8c5dd3f640 1 --2- 172.21.15.60:0/4113366641 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8c3c0762d0 0x7f8c3c078790 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8c580cd930 tx=0x7f8c48009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:33.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:33.967+0000 7f8c4effd640 1 -- 172.21.15.60:0/4113366641 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8c5404b030 con 0x7f8c580723a0 2024-05-09T05:01:34.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.161+0000 7f8c607a4640 1 -- 172.21.15.60:0/4113366641 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f8c58160bc0 con 0x7f8c580723a0 2024-05-09T05:01:34.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.164+0000 7f8c4effd640 1 -- 172.21.15.60:0/4113366641 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v15)=0 v15) ==== 122+0+0 (secure 0 0 0) 0x7f8c540817a0 con 0x7f8c580723a0 2024-05-09T05:01:34.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.166+0000 7f8c607a4640 1 -- 172.21.15.60:0/4113366641 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8c3c0762d0 msgr2=0x7f8c3c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:34.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.166+0000 7f8c607a4640 1 --2- 172.21.15.60:0/4113366641 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8c3c0762d0 0x7f8c3c078790 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8c580cd930 tx=0x7f8c48009040 comp rx=0 tx=0).stop 2024-05-09T05:01:34.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.166+0000 7f8c607a4640 1 -- 172.21.15.60:0/4113366641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c580723a0 msgr2=0x7f8c580ae7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:34.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.166+0000 7f8c607a4640 1 --2- 172.21.15.60:0/4113366641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c580723a0 0x7f8c580ae7d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8c54000c00 tx=0x7f8c540046c0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.166+0000 7f8c607a4640 1 -- 172.21.15.60:0/4113366641 shutdown_connections 2024-05-09T05:01:34.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.166+0000 7f8c607a4640 1 --2- 172.21.15.60:0/4113366641 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8c3c0762d0 0x7f8c3c078790 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.166+0000 7f8c607a4640 1 --2- 172.21.15.60:0/4113366641 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8c5815d910 0x7f8c580cd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.166+0000 7f8c607a4640 1 --2- 172.21.15.60:0/4113366641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c58072780 0x7f8c580ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.166+0000 7f8c607a4640 1 --2- 172.21.15.60:0/4113366641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c580723a0 0x7f8c580ae7d0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.166+0000 7f8c607a4640 1 -- 172.21.15.60:0/4113366641 >> 172.21.15.60:0/4113366641 conn(0x7f8c58070820 msgr2=0x7f8c580d1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:34.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.166+0000 7f8c607a4640 1 -- 172.21.15.60:0/4113366641 shutdown_connections 2024-05-09T05:01:34.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.166+0000 7f8c607a4640 1 -- 172.21.15.60:0/4113366641 wait complete. 2024-05-09T05:01:34.178 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T05:01:34.179 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T05:01:34.220 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T05:01:34.259 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T05:01:34.259 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:01:34.259 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:01:34.276 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057102,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498908k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19678 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498908k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T05:01:34.277 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T05:01:34.277 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:01:34.316 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056322,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497656k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248824k,nr_inodes=812206,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497656k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T05:01:34.316 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:01:34.316 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:01:34.332 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057102,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498908k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19678 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498908k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T05:01:34.332 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T05:01:34.332 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:01:34.372 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056322,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497656k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248824k,nr_inodes=812206,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497656k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T05:01:34.372 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-09T05:01:34.372 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:34.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.476+0000 7f1356322640 1 Processor -- start 2024-05-09T05:01:34.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.476+0000 7f1356322640 1 -- start start 2024-05-09T05:01:34.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.477+0000 7f1356322640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13500d1050 0x7f13500ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:34.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.477+0000 7f1356322640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1350096eb0 0x7f135009b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:34.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.477+0000 7f1356322640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13500af370 con 0x7f135009be90 2024-05-09T05:01:34.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.477+0000 7f1356322640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f1350096a00 con 0x7f1350096eb0 2024-05-09T05:01:34.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.477+0000 7f1356322640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f13500d2dd0 con 0x7f13500d1050 2024-05-09T05:01:34.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.477+0000 7f1347fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1350096eb0 0x7f135009b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:34.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.477+0000 7f1347fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1350096eb0 0x7f135009b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41950/0 (socket says 172.21.15.60:41950) 2024-05-09T05:01:34.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.477+0000 7f1347fff640 1 -- 172.21.15.60:0/3961395180 learned_addr learned my addr 172.21.15.60:0/3961395180 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:34.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.477+0000 7f134ffff640 1 --2- 172.21.15.60:0/3961395180 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13500d1050 0x7f13500ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:34.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.478+0000 7f1347fff640 1 -- 172.21.15.60:0/3961395180 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13500d1050 msgr2=0x7f13500ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:34.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.478+0000 7f1347fff640 1 --2- 172.21.15.60:0/3961395180 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13500d1050 0x7f13500ae9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.478+0000 7f1347fff640 1 -- 172.21.15.60:0/3961395180 >> v1:172.21.15.114:6789/0 conn(0x7f135009be90 legacy=0x7f13500d1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:34.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.478+0000 7f1347fff640 1 -- 172.21.15.60:0/3961395180 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13500d30b0 con 0x7f1350096eb0 2024-05-09T05:01:34.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.478+0000 7f1347fff640 1 --2- 172.21.15.60:0/3961395180 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1350096eb0 0x7f135009b950 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1338009870 tx=0x7f133802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72b33000d95e2130 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:34.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.478+0000 7f134f7fe640 1 -- 172.21.15.60:0/3961395180 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1338037070 con 0x7f1350096eb0 2024-05-09T05:01:34.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.478+0000 7f134f7fe640 1 -- 172.21.15.60:0/3961395180 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1338032440 con 0x7f1350096eb0 2024-05-09T05:01:34.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.478+0000 7f134f7fe640 1 -- 172.21.15.60:0/3961395180 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1338032810 con 0x7f1350096eb0 2024-05-09T05:01:34.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.479+0000 7f1356322640 1 -- 172.21.15.60:0/3961395180 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1350096eb0 msgr2=0x7f135009b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:34.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.479+0000 7f1356322640 1 --2- 172.21.15.60:0/3961395180 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1350096eb0 0x7f135009b950 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1338009870 tx=0x7f133802a270 comp rx=0 tx=0).stop 2024-05-09T05:01:34.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.479+0000 7f1356322640 1 -- 172.21.15.60:0/3961395180 shutdown_connections 2024-05-09T05:01:34.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.479+0000 7f1356322640 1 --2- 172.21.15.60:0/3961395180 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1350096eb0 0x7f135009b950 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.479+0000 7f1356322640 1 --2- 172.21.15.60:0/3961395180 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f13500d1050 0x7f13500ae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.479+0000 7f1356322640 1 -- 172.21.15.60:0/3961395180 >> 172.21.15.60:0/3961395180 conn(0x7f13500993f0 msgr2=0x7f1350099800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:34.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.479+0000 7f1356322640 1 -- 172.21.15.60:0/3961395180 shutdown_connections 2024-05-09T05:01:34.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.479+0000 7f1356322640 1 -- 172.21.15.60:0/3961395180 wait complete. 2024-05-09T05:01:34.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.480+0000 7f1356322640 1 Processor -- start 2024-05-09T05:01:34.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.480+0000 7f1356322640 1 -- start start 2024-05-09T05:01:34.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.481+0000 7f1356322640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1350096eb0 0x7f13501e6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:34.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.481+0000 7f1356322640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f135009be90 0x7f13501ea620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:34.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.481+0000 7f1356322640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13500d1050 0x7f135006e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:34.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.481+0000 7f1356322640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f13500af0b0 con 0x7f1350096eb0 2024-05-09T05:01:34.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.481+0000 7f1356322640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f13500aef30 con 0x7f13500d1050 2024-05-09T05:01:34.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.481+0000 7f1356322640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f13500d1b40 con 0x7f135009be90 2024-05-09T05:01:34.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.481+0000 7f134ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1350096eb0 0x7f13501e6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:34.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.481+0000 7f134ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1350096eb0 0x7f13501e6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41962/0 (socket says 172.21.15.60:41962) 2024-05-09T05:01:34.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.481+0000 7f134ffff640 1 -- 172.21.15.60:0/2798880636 learned_addr learned my addr 172.21.15.60:0/2798880636 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:34.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.481+0000 7f1347fff640 1 --2- 172.21.15.60:0/2798880636 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f135009be90 0x7f13501ea620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:34.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.481+0000 7f13548bf640 1 --2- 172.21.15.60:0/2798880636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13500d1050 0x7f135006e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:34.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.481+0000 7f134ffff640 1 -- 172.21.15.60:0/2798880636 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f135009be90 msgr2=0x7f13501ea620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:34.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.481+0000 7f134ffff640 1 --2- 172.21.15.60:0/2798880636 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f135009be90 0x7f13501ea620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.481+0000 7f134ffff640 1 -- 172.21.15.60:0/2798880636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13500d1050 msgr2=0x7f135006e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:34.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.481+0000 7f134ffff640 1 --2- 172.21.15.60:0/2798880636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13500d1050 0x7f135006e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.481+0000 7f134ffff640 1 -- 172.21.15.60:0/2798880636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13501e7250 con 0x7f1350096eb0 2024-05-09T05:01:34.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.481+0000 7f1347fff640 1 --2- 172.21.15.60:0/2798880636 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f135009be90 0x7f13501ea620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:34.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.482+0000 7f13548bf640 1 --2- 172.21.15.60:0/2798880636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13500d1050 0x7f135006e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:34.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.482+0000 7f134ffff640 1 --2- 172.21.15.60:0/2798880636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1350096eb0 0x7f13501e6d10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f134000e9b0 tx=0x7f134000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:34.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.482+0000 7f134d7fa640 1 -- 172.21.15.60:0/2798880636 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1340017070 con 0x7f1350096eb0 2024-05-09T05:01:34.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.482+0000 7f134d7fa640 1 -- 172.21.15.60:0/2798880636 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f134000c9a0 con 0x7f1350096eb0 2024-05-09T05:01:34.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.482+0000 7f1356322640 1 -- 172.21.15.60:0/2798880636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13501e92a0 con 0x7f1350096eb0 2024-05-09T05:01:34.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.482+0000 7f134d7fa640 1 -- 172.21.15.60:0/2798880636 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134000cd90 con 0x7f1350096eb0 2024-05-09T05:01:34.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.482+0000 7f1356322640 1 -- 172.21.15.60:0/2798880636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13501e9860 con 0x7f1350096eb0 2024-05-09T05:01:34.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.483+0000 7f1356322640 1 -- 172.21.15.60:0/2798880636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1350075540 con 0x7f1350096eb0 2024-05-09T05:01:34.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.483+0000 7f134d7fa640 1 -- 172.21.15.60:0/2798880636 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f134001a070 con 0x7f1350096eb0 2024-05-09T05:01:34.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.484+0000 7f134d7fa640 1 --2- 172.21.15.60:0/2798880636 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f13300762d0 0x7f1330078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:34.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.484+0000 7f134d7fa640 1 -- 172.21.15.60:0/2798880636 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5335+0+0 (secure 0 0 0) 0x7f13400974d0 con 0x7f1350096eb0 2024-05-09T05:01:34.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.484+0000 7f1347fff640 1 --2- 172.21.15.60:0/2798880636 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f13300762d0 0x7f1330078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:34.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.485+0000 7f1347fff640 1 --2- 172.21.15.60:0/2798880636 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f13300762d0 0x7f1330078790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1338034000 tx=0x7f133802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:34.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.488+0000 7f134d7fa640 1 -- 172.21.15.60:0/2798880636 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1340060fa0 con 0x7f1350096eb0 2024-05-09T05:01:34.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.719+0000 7f1356322640 1 -- 172.21.15.60:0/2798880636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1350098ae0 con 0x7f1350096eb0 2024-05-09T05:01:34.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.719+0000 7f134d7fa640 1 -- 172.21.15.60:0/2798880636 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+3872 (secure 0 0 0) 0x7f13400608a0 con 0x7f1350096eb0 2024-05-09T05:01:34.720 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:34.720 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4547,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/2372252858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":2372252858},{"type":"v1","addr":"172.21.15.114:6836","nonce":2372252858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1761282610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1761282610},{"type":"v1","addr":"172.21.15.114:6837","nonce":1761282610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":4765,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/300408012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":300408012},{"type":"v1","addr":"172.21.15.60:6835","nonce":300408012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5313,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/1368786916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1368786916},{"type":"v1","addr":"172.21.15.60:6834","nonce":1368786916}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[]} 2024-05-09T05:01:34.720 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 20 2024-05-09T05:01:34.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.721+0000 7f1356322640 1 -- 172.21.15.60:0/2798880636 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f13300762d0 msgr2=0x7f1330078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:34.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.721+0000 7f1356322640 1 --2- 172.21.15.60:0/2798880636 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f13300762d0 0x7f1330078790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1338034000 tx=0x7f133802e000 comp rx=0 tx=0).stop 2024-05-09T05:01:34.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.721+0000 7f1356322640 1 -- 172.21.15.60:0/2798880636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1350096eb0 msgr2=0x7f13501e6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:34.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.721+0000 7f1356322640 1 --2- 172.21.15.60:0/2798880636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1350096eb0 0x7f13501e6d10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f134000e9b0 tx=0x7f134000ee70 comp rx=0 tx=0).stop 2024-05-09T05:01:34.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.722+0000 7f1356322640 1 -- 172.21.15.60:0/2798880636 shutdown_connections 2024-05-09T05:01:34.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.722+0000 7f1356322640 1 --2- 172.21.15.60:0/2798880636 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f13300762d0 0x7f1330078790 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.722+0000 7f1356322640 1 --2- 172.21.15.60:0/2798880636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13500d1050 0x7f135006e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.722+0000 7f1356322640 1 --2- 172.21.15.60:0/2798880636 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f135009be90 0x7f13501ea620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.722+0000 7f1356322640 1 --2- 172.21.15.60:0/2798880636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1350096eb0 0x7f13501e6d10 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.722+0000 7f1356322640 1 -- 172.21.15.60:0/2798880636 >> 172.21.15.60:0/2798880636 conn(0x7f13500993f0 msgr2=0x7f135009a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:34.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.722+0000 7f1356322640 1 -- 172.21.15.60:0/2798880636 shutdown_connections 2024-05-09T05:01:34.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.722+0000 7f1356322640 1 -- 172.21.15.60:0/2798880636 wait complete. 2024-05-09T05:01:34.733 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-09T05:01:34.734 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-09T05:01:34.734 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-09T05:01:34.734 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:01:34.734 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-09T05:01:34.734 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-09T05:01:34.735 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-09T05:01:34.735 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:01:34.735 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-09T05:01:34.735 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-09T05:01:34.735 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-09T05:01:34.735 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:01:34.735 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-09T05:01:34.736 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-09T05:01:34.736 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-09T05:01:34.736 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:01:34.743 INFO:tasks.ceph.mds.a:Stopped 2024-05-09T05:01:34.744 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-09T05:01:34.744 INFO:tasks.ceph.mds.c:Stopped 2024-05-09T05:01:34.745 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-09T05:01:34.746 INFO:tasks.ceph.mds.b:Stopped 2024-05-09T05:01:34.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-09T05:01:34.747 INFO:tasks.ceph.mds.d:Stopped 2024-05-09T05:01:34.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-09T05:01:34.787 INFO:tasks.ceph.mds.a:Started 2024-05-09T05:01:34.787 DEBUG:teuthology.parallel:result is None 2024-05-09T05:01:34.788 INFO:tasks.ceph.mds.c:Started 2024-05-09T05:01:34.788 DEBUG:teuthology.parallel:result is None 2024-05-09T05:01:34.789 INFO:tasks.ceph.mds.b:Started 2024-05-09T05:01:34.789 DEBUG:teuthology.parallel:result is None 2024-05-09T05:01:34.789 INFO:tasks.ceph.mds.d:Started 2024-05-09T05:01:34.790 DEBUG:teuthology.parallel:result is None 2024-05-09T05:01:34.790 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-09T05:01:34.866 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2024-05-09T05:01:34.867 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2024-05-09T05:01:34.869 INFO:tasks.ceph.mds.b.smithi114.stdout:starting mds.b at 2024-05-09T05:01:34.869 INFO:tasks.ceph.mds.d.smithi114.stdout:starting mds.d at 2024-05-09T05:01:34.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.898+0000 7fe19102c640 1 Processor -- start 2024-05-09T05:01:34.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.898+0000 7fe19102c640 1 -- start start 2024-05-09T05:01:34.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.899+0000 7fe19102c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe18c06f9f0 0x7fe18c06ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:34.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.899+0000 7fe19102c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe18c06ff60 con 0x7fe18c1606f0 2024-05-09T05:01:34.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.899+0000 7fe19102c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe18c070240 con 0x7fe18c15f300 2024-05-09T05:01:34.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.899+0000 7fe19102c640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fe18c070520 con 0x7fe18c06f9f0 2024-05-09T05:01:34.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.899+0000 7fe183fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe18c1606f0 0x7fe18c0d33f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42320/0 (socket says 172.21.15.60:42320) 2024-05-09T05:01:34.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.899+0000 7fe183fff640 1 -- 172.21.15.60:0/949147302 learned_addr learned my addr 172.21.15.60:0/949147302 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:34.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.899+0000 7fe18a59c640 1 -- 172.21.15.60:0/949147302 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3393823383 0 0) 0x7fe18c06ff60 con 0x7fe18c1606f0 2024-05-09T05:01:34.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.899+0000 7fe18a59c640 1 -- 172.21.15.60:0/949147302 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe178003520 con 0x7fe18c1606f0 2024-05-09T05:01:34.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.899+0000 7fe18b59e640 1 --2- 172.21.15.60:0/949147302 >> v2:172.21.15.114:3300/0 conn(0x7fe18c06f9f0 0x7fe18c06ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:34.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18a59c640 1 -- 172.21.15.60:0/949147302 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3456868657 0 0) 0x7fe178003520 con 0x7fe18c1606f0 2024-05-09T05:01:34.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18a59c640 1 -- 172.21.15.60:0/949147302 >> v2:172.21.15.114:3300/0 conn(0x7fe18c06f9f0 msgr2=0x7fe18c06ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:34.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18a59c640 1 --2- 172.21.15.60:0/949147302 >> v2:172.21.15.114:3300/0 conn(0x7fe18c06f9f0 0x7fe18c06ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18a59c640 1 -- 172.21.15.60:0/949147302 >> v1:172.21.15.114:6789/0 conn(0x7fe18c15f300 legacy=0x7fe18c1639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:34.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18a59c640 1 -- 172.21.15.60:0/949147302 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe18c06e740 con 0x7fe18c1606f0 2024-05-09T05:01:34.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18a59c640 1 -- 172.21.15.60:0/949147302 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7fe1740016f0 con 0x7fe18c1606f0 2024-05-09T05:01:34.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18a59c640 1 -- 172.21.15.60:0/949147302 >> v1:172.21.15.60:6789/0 conn(0x7fe18c1606f0 legacy=0x7fe18c0d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:34.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18a59c640 1 --2- 172.21.15.60:0/949147302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe178003a30 0x7fe178003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:34.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18a59c640 1 --2- 172.21.15.60:0/949147302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe178004800 0x7fe178004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:34.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18a59c640 1 --2- 172.21.15.60:0/949147302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1780051c0 0x7fe17800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:34.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18a59c640 1 -- 172.21.15.60:0/949147302 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c070520 con 0x7fe178003a30 2024-05-09T05:01:34.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18a59c640 1 -- 172.21.15.60:0/949147302 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe17800c0c0 con 0x7fe1780051c0 2024-05-09T05:01:34.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18a59c640 1 -- 172.21.15.60:0/949147302 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe17800c2c0 con 0x7fe178004800 2024-05-09T05:01:34.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18ad9d640 1 --2- 172.21.15.60:0/949147302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe178003a30 0x7fe178003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:34.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe183fff640 1 --2- 172.21.15.60:0/949147302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe178004800 0x7fe178004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:34.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18b59e640 1 --2- 172.21.15.60:0/949147302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1780051c0 0x7fe17800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:34.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18ad9d640 1 -- 172.21.15.60:0/949147302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe178004800 msgr2=0x7fe178004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:34.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18ad9d640 1 --2- 172.21.15.60:0/949147302 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe178004800 0x7fe178004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18ad9d640 1 -- 172.21.15.60:0/949147302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1780051c0 msgr2=0x7fe17800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:34.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18ad9d640 1 --2- 172.21.15.60:0/949147302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1780051c0 0x7fe17800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18ad9d640 1 -- 172.21.15.60:0/949147302 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe178003520 con 0x7fe178003a30 2024-05-09T05:01:34.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18b59e640 1 --2- 172.21.15.60:0/949147302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1780051c0 0x7fe17800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:34.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.900+0000 7fe18b59e640 1 -- 172.21.15.60:0/949147302 reap_dead start 2024-05-09T05:01:34.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.901+0000 7fe18ad9d640 1 --2- 172.21.15.60:0/949147302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe178003a30 0x7fe178003e10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe17c001810 tx=0x7fe178005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9e5ca82d326ae3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:34.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.901+0000 7fe18a59c640 1 -- 172.21.15.60:0/949147302 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c037070 con 0x7fe178003a30 2024-05-09T05:01:34.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.901+0000 7fe18a59c640 1 -- 172.21.15.60:0/949147302 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe17c02ac30 con 0x7fe178003a30 2024-05-09T05:01:34.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.901+0000 7fe18a59c640 1 -- 172.21.15.60:0/949147302 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c001430 con 0x7fe178003a30 2024-05-09T05:01:34.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.901+0000 7fe19102c640 1 -- 172.21.15.60:0/949147302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe178003a30 msgr2=0x7fe178003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:34.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.901+0000 7fe19102c640 1 --2- 172.21.15.60:0/949147302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe178003a30 0x7fe178003e10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe17c001810 tx=0x7fe178005830 comp rx=0 tx=0).stop 2024-05-09T05:01:34.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.902+0000 7fe19102c640 1 -- 172.21.15.60:0/949147302 shutdown_connections 2024-05-09T05:01:34.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.902+0000 7fe19102c640 1 --2- 172.21.15.60:0/949147302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe178003a30 0x7fe178003e10 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.902+0000 7fe19102c640 1 -- 172.21.15.60:0/949147302 >> 172.21.15.60:0/949147302 conn(0x7fe18c09b850 msgr2=0x7fe18c09bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:34.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.902+0000 7fe19102c640 1 -- 172.21.15.60:0/949147302 shutdown_connections 2024-05-09T05:01:34.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.902+0000 7fe19102c640 1 -- 172.21.15.60:0/949147302 wait complete. 2024-05-09T05:01:34.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.903+0000 7fe19102c640 1 Processor -- start 2024-05-09T05:01:34.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.903+0000 7fe19102c640 1 -- start start 2024-05-09T05:01:34.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.904+0000 7fe19102c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe178003a30 0x7fe18c098e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:34.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.904+0000 7fe19102c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe18c09f0c0 0x7fe18c1f4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:34.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.904+0000 7fe19102c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe18c095540 0x7fe18c0959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:34.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.904+0000 7fe19102c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c163b70 con 0x7fe178003a30 2024-05-09T05:01:34.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.904+0000 7fe19102c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c1639f0 con 0x7fe18c095540 2024-05-09T05:01:34.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.904+0000 7fe19102c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe18c163cf0 con 0x7fe18c09f0c0 2024-05-09T05:01:34.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.904+0000 7fe18ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe178003a30 0x7fe18c098e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:34.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.904+0000 7fe18ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe178003a30 0x7fe18c098e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42016/0 (socket says 172.21.15.60:42016) 2024-05-09T05:01:34.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.904+0000 7fe18ad9d640 1 -- 172.21.15.60:0/2968258964 learned_addr learned my addr 172.21.15.60:0/2968258964 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:34.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.904+0000 7fe183fff640 1 --2- 172.21.15.60:0/2968258964 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe18c09f0c0 0x7fe18c1f4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:34.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.904+0000 7fe18b59e640 1 --2- 172.21.15.60:0/2968258964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe18c095540 0x7fe18c0959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:34.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.904+0000 7fe18ad9d640 1 -- 172.21.15.60:0/2968258964 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe18c09f0c0 msgr2=0x7fe18c1f4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:34.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.904+0000 7fe18ad9d640 1 --2- 172.21.15.60:0/2968258964 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe18c09f0c0 0x7fe18c1f4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.904+0000 7fe18ad9d640 1 -- 172.21.15.60:0/2968258964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe18c095540 msgr2=0x7fe18c0959c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:34.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.904+0000 7fe18ad9d640 1 --2- 172.21.15.60:0/2968258964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe18c095540 0x7fe18c0959c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:34.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.904+0000 7fe18ad9d640 1 -- 172.21.15.60:0/2968258964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe18c099410 con 0x7fe178003a30 2024-05-09T05:01:34.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.904+0000 7fe18ad9d640 1 --2- 172.21.15.60:0/2968258964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe178003a30 0x7fe18c098e30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe17c000fb0 tx=0x7fe17c001110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:34.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.904+0000 7fe1837fe640 1 -- 172.21.15.60:0/2968258964 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c037050 con 0x7fe178003a30 2024-05-09T05:01:34.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.905+0000 7fe1837fe640 1 -- 172.21.15.60:0/2968258964 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe17c043070 con 0x7fe178003a30 2024-05-09T05:01:34.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.905+0000 7fe19102c640 1 -- 172.21.15.60:0/2968258964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe18c093690 con 0x7fe178003a30 2024-05-09T05:01:34.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.905+0000 7fe1837fe640 1 -- 172.21.15.60:0/2968258964 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c007a10 con 0x7fe178003a30 2024-05-09T05:01:34.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.905+0000 7fe19102c640 1 -- 172.21.15.60:0/2968258964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe18c1f52e0 con 0x7fe178003a30 2024-05-09T05:01:34.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.906+0000 7fe19102c640 1 -- 172.21.15.60:0/2968258964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe18c051040 con 0x7fe178003a30 2024-05-09T05:01:34.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.906+0000 7fe1837fe640 1 -- 172.21.15.60:0/2968258964 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe17c03e650 con 0x7fe178003a30 2024-05-09T05:01:34.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.907+0000 7fe1837fe640 1 --2- 172.21.15.60:0/2968258964 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe16c0762d0 0x7fe16c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:34.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.907+0000 7fe1837fe640 1 -- 172.21.15.60:0/2968258964 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5335+0+0 (secure 0 0 0) 0x7fe17c0b7ab0 con 0x7fe178003a30 2024-05-09T05:01:34.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.907+0000 7fe183fff640 1 --2- 172.21.15.60:0/2968258964 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe16c0762d0 0x7fe16c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:34.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.907+0000 7fe183fff640 1 --2- 172.21.15.60:0/2968258964 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe16c0762d0 0x7fe16c078790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0b08c0 tx=0x7fe1740078f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:34.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:34.911+0000 7fe1837fe640 1 -- 172.21.15.60:0/2968258964 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe17c04d020 con 0x7fe178003a30 2024-05-09T05:01:35.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.114+0000 7fe19102c640 1 -- 172.21.15.60:0/2968258964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fe18c073f90 con 0x7fe178003a30 2024-05-09T05:01:35.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.836+0000 7fe1837fe640 1 -- 172.21.15.60:0/2968258964 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v45) ==== 97+0+0 (secure 0 0 0) 0x7fe17c080f30 con 0x7fe178003a30 2024-05-09T05:01:35.837 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blocklist entries 2024-05-09T05:01:35.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.840+0000 7fe19102c640 1 -- 172.21.15.60:0/2968258964 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe16c0762d0 msgr2=0x7fe16c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:35.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.840+0000 7fe19102c640 1 --2- 172.21.15.60:0/2968258964 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe16c0762d0 0x7fe16c078790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0b08c0 tx=0x7fe1740078f0 comp rx=0 tx=0).stop 2024-05-09T05:01:35.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.840+0000 7fe19102c640 1 -- 172.21.15.60:0/2968258964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe178003a30 msgr2=0x7fe18c098e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:35.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.840+0000 7fe19102c640 1 --2- 172.21.15.60:0/2968258964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe178003a30 0x7fe18c098e30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe17c000fb0 tx=0x7fe17c001110 comp rx=0 tx=0).stop 2024-05-09T05:01:35.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.840+0000 7fe19102c640 1 -- 172.21.15.60:0/2968258964 shutdown_connections 2024-05-09T05:01:35.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.840+0000 7fe19102c640 1 --2- 172.21.15.60:0/2968258964 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe16c0762d0 0x7fe16c078790 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:35.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.840+0000 7fe19102c640 1 --2- 172.21.15.60:0/2968258964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe18c095540 0x7fe18c0959c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:35.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.840+0000 7fe19102c640 1 --2- 172.21.15.60:0/2968258964 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe18c09f0c0 0x7fe18c1f4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:35.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.840+0000 7fe19102c640 1 --2- 172.21.15.60:0/2968258964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe178003a30 0x7fe18c098e30 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:35.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.840+0000 7fe19102c640 1 -- 172.21.15.60:0/2968258964 >> 172.21.15.60:0/2968258964 conn(0x7fe18c09b850 msgr2=0x7fe18c0d3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:35.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.840+0000 7fe19102c640 1 -- 172.21.15.60:0/2968258964 shutdown_connections 2024-05-09T05:01:35.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.840+0000 7fe19102c640 1 -- 172.21.15.60:0/2968258964 wait complete. 2024-05-09T05:01:35.856 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-09T05:01:35.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.966+0000 7f52e2124640 1 Processor -- start 2024-05-09T05:01:35.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.966+0000 7f52e2124640 1 -- start start 2024-05-09T05:01:35.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.967+0000 7f52e2124640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f52dc15f3e0 0x7f52dc0b4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:35.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.967+0000 7f52e2124640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52dc073310 0x7f52dc0b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:35.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.967+0000 7f52e2124640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52dc071f30 con 0x7f52dc09c8b0 2024-05-09T05:01:35.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.967+0000 7f52e2124640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f52dc071ce0 con 0x7f52dc15f3e0 2024-05-09T05:01:35.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.967+0000 7f52e2124640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f52dc072210 con 0x7f52dc073310 2024-05-09T05:01:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.967+0000 7f52db7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f52dc15f3e0 0x7f52dc0b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.967+0000 7f52dbfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52dc073310 0x7f52dc0b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.967+0000 7f52dbfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52dc073310 0x7f52dc0b4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.60:49390/0 (socket says 172.21.15.60:49390) 2024-05-09T05:01:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.967+0000 7f52dbfff640 1 -- 172.21.15.60:0/4161238512 learned_addr learned my addr 172.21.15.60:0/4161238512 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.967+0000 7f52db7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f52dc15f3e0 0x7f52dc0b4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.60:54720/0 (socket says 172.21.15.60:54720) 2024-05-09T05:01:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.967+0000 7f52daffd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f52dc09c8b0 0x7f52dc06e770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.60:35512/0 (socket says 172.21.15.60:35512) 2024-05-09T05:01:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.968+0000 7f52dbfff640 1 -- 172.21.15.60:0/4161238512 >> v2:172.21.15.114:3300/0 conn(0x7f52dc15f3e0 msgr2=0x7f52dc0b4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.968+0000 7f52dbfff640 1 --2- 172.21.15.60:0/4161238512 >> v2:172.21.15.114:3300/0 conn(0x7f52dc15f3e0 0x7f52dc0b4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.968+0000 7f52dbfff640 1 -- 172.21.15.60:0/4161238512 >> v1:172.21.15.114:6789/0 conn(0x7f52dc09c8b0 legacy=0x7f52dc06e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.968+0000 7f52dbfff640 1 -- 172.21.15.60:0/4161238512 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52dc0724f0 con 0x7f52dc073310 2024-05-09T05:01:35.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.968+0000 7f52dbfff640 1 --2- 172.21.15.60:0/4161238512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52dc073310 0x7f52dc0b4b70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f52d4009e50 tx=0x7f52d402a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=369770d546849983 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:35.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.969+0000 7f52da7fc640 1 -- 172.21.15.60:0/4161238512 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d4036070 con 0x7f52dc073310 2024-05-09T05:01:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.969+0000 7f52da7fc640 1 -- 172.21.15.60:0/4161238512 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52d4031600 con 0x7f52dc073310 2024-05-09T05:01:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.969+0000 7f52da7fc640 1 -- 172.21.15.60:0/4161238512 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d40319d0 con 0x7f52dc073310 2024-05-09T05:01:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.969+0000 7f52e2124640 1 -- 172.21.15.60:0/4161238512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52dc073310 msgr2=0x7f52dc0b4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.969+0000 7f52e2124640 1 --2- 172.21.15.60:0/4161238512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52dc073310 0x7f52dc0b4b70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f52d4009e50 tx=0x7f52d402a680 comp rx=0 tx=0).stop 2024-05-09T05:01:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.969+0000 7f52e2124640 1 -- 172.21.15.60:0/4161238512 shutdown_connections 2024-05-09T05:01:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.969+0000 7f52e2124640 1 --2- 172.21.15.60:0/4161238512 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52dc073310 0x7f52dc0b4b70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.969+0000 7f52e2124640 1 --2- 172.21.15.60:0/4161238512 >> v2:172.21.15.114:3300/0 conn(0x7f52dc15f3e0 0x7f52dc0b4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.969+0000 7f52e2124640 1 -- 172.21.15.60:0/4161238512 >> 172.21.15.60:0/4161238512 conn(0x7f52dc0d1470 msgr2=0x7f52dc0d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.969+0000 7f52e2124640 1 -- 172.21.15.60:0/4161238512 shutdown_connections 2024-05-09T05:01:35.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.969+0000 7f52e2124640 1 -- 172.21.15.60:0/4161238512 wait complete. 2024-05-09T05:01:35.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.970+0000 7f52e2124640 1 Processor -- start 2024-05-09T05:01:35.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.971+0000 7f52e2124640 1 -- start start 2024-05-09T05:01:35.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.971+0000 7f52e2124640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52dc073310 0x7f52dc0d4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:35.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.971+0000 7f52e2124640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52dc09c8b0 0x7f52dc15f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:35.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.971+0000 7f52e2124640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52dc15f3e0 0x7f52dc0ac390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:35.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.971+0000 7f52e2124640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f52dc15f7c0 con 0x7f52dc09c8b0 2024-05-09T05:01:35.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.971+0000 7f52e2124640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f52dc072ab0 con 0x7f52dc073310 2024-05-09T05:01:35.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.971+0000 7f52e2124640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f52dc09f110 con 0x7f52dc15f3e0 2024-05-09T05:01:35.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.972+0000 7f52daffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52dc09c8b0 0x7f52dc15f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:35.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.972+0000 7f52daffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52dc09c8b0 0x7f52dc15f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42032/0 (socket says 172.21.15.60:42032) 2024-05-09T05:01:35.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.972+0000 7f52daffd640 1 -- 172.21.15.60:0/2745519400 learned_addr learned my addr 172.21.15.60:0/2745519400 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:35.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.972+0000 7f52db7fe640 1 --2- 172.21.15.60:0/2745519400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52dc073310 0x7f52dc0d4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:35.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.972+0000 7f52dbfff640 1 --2- 172.21.15.60:0/2745519400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52dc15f3e0 0x7f52dc0ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:35.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.972+0000 7f52daffd640 1 -- 172.21.15.60:0/2745519400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52dc15f3e0 msgr2=0x7f52dc0ac390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:35.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.972+0000 7f52daffd640 1 --2- 172.21.15.60:0/2745519400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52dc15f3e0 0x7f52dc0ac390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:35.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.972+0000 7f52daffd640 1 -- 172.21.15.60:0/2745519400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52dc073310 msgr2=0x7f52dc0d4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:35.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.972+0000 7f52daffd640 1 --2- 172.21.15.60:0/2745519400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52dc073310 0x7f52dc0d4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:35.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.972+0000 7f52daffd640 1 -- 172.21.15.60:0/2745519400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52dc0d3ac0 con 0x7f52dc09c8b0 2024-05-09T05:01:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.972+0000 7f52daffd640 1 --2- 172.21.15.60:0/2745519400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52dc09c8b0 0x7f52dc15f940 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f52d000c450 tx=0x7f52d000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.972+0000 7f52c3fff640 1 -- 172.21.15.60:0/2745519400 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d0018070 con 0x7f52dc09c8b0 2024-05-09T05:01:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.972+0000 7f52c3fff640 1 -- 172.21.15.60:0/2745519400 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52d0007480 con 0x7f52dc09c8b0 2024-05-09T05:01:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.972+0000 7f52c3fff640 1 -- 172.21.15.60:0/2745519400 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d00047e0 con 0x7f52dc09c8b0 2024-05-09T05:01:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.972+0000 7f52e2124640 1 -- 172.21.15.60:0/2745519400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52dc0d3da0 con 0x7f52dc09c8b0 2024-05-09T05:01:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.973+0000 7f52e2124640 1 -- 172.21.15.60:0/2745519400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52dc0d4360 con 0x7f52dc09c8b0 2024-05-09T05:01:35.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.973+0000 7f52e2124640 1 -- 172.21.15.60:0/2745519400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52dc051040 con 0x7f52dc09c8b0 2024-05-09T05:01:35.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.974+0000 7f52c3fff640 1 -- 172.21.15.60:0/2745519400 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f52d001b070 con 0x7f52dc09c8b0 2024-05-09T05:01:35.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.975+0000 7f52c3fff640 1 --2- 172.21.15.60:0/2745519400 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f52c80762d0 0x7f52c8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:35.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.975+0000 7f52c3fff640 1 -- 172.21.15.60:0/2745519400 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5249+0+0 (secure 0 0 0) 0x7f52d0096650 con 0x7f52dc09c8b0 2024-05-09T05:01:35.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.975+0000 7f52db7fe640 1 --2- 172.21.15.60:0/2745519400 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f52c80762d0 0x7f52c8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:35.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.976+0000 7f52db7fe640 1 --2- 172.21.15.60:0/2745519400 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f52c80762d0 0x7f52c8078790 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f52cc009b40 tx=0x7f52cc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:35.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:35.979+0000 7f52c3fff640 1 -- 172.21.15.60:0/2745519400 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f52d005fa80 con 0x7f52dc09c8b0 2024-05-09T05:01:36.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.246+0000 7f52e2124640 1 -- 172.21.15.60:0/2745519400 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f52dc0d18f0 con 0x7f52dc09c8b0 2024-05-09T05:01:36.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.246+0000 7f52c3fff640 1 -- 172.21.15.60:0/2745519400 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) ==== 80+0+5304 (secure 0 0 0) 0x7f52d00680c0 con 0x7f52dc09c8b0 2024-05-09T05:01:36.247 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:36.247 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:01:36.247 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2024-05-09T05:01:36.247 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.247 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2024-05-09T05:01:36.247 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAsWDxmJBQ1LRAAJfklX7rdjYuv6WIp+8rgMg==", 2024-05-09T05:01:36.247 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:01:36.247 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2024-05-09T05:01:36.248 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2024-05-09T05:01:36.248 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2024-05-09T05:01:36.248 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:01:36.248 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:36.248 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.248 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2024-05-09T05:01:36.248 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAsWDxmUN0fORAAaKZA9GlWlOfJaY/SYAXNlw==", 2024-05-09T05:01:36.248 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:01:36.248 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2024-05-09T05:01:36.248 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2024-05-09T05:01:36.249 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2024-05-09T05:01:36.249 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:01:36.249 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:36.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.249 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.c", 2024-05-09T05:01:36.249 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAsWDxmmr9iNBAAyatU9+hjHFV/AFTKpVxMPQ==", 2024-05-09T05:01:36.249 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:01:36.249 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2024-05-09T05:01:36.249 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2024-05-09T05:01:36.249 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2024-05-09T05:01:36.249 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:01:36.249 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:36.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.250 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.d", 2024-05-09T05:01:36.250 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAtWDxms0CIBBAAsHfza+C91qgfGjrczfoGeg==", 2024-05-09T05:01:36.250 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:01:36.250 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2024-05-09T05:01:36.250 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2024-05-09T05:01:36.250 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2024-05-09T05:01:36.250 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:01:36.250 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:36.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.251 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2024-05-09T05:01:36.251 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAuWDxmOInrKBAAthThp6oxwvbii2eif7krkg==", 2024-05-09T05:01:36.251 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:01:36.251 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2024-05-09T05:01:36.251 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2024-05-09T05:01:36.251 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:01:36.251 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:36.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.251 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2024-05-09T05:01:36.251 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAxWDxmbDifJxAAhJ5xGS3YfkNkYuJYF7MiIg==", 2024-05-09T05:01:36.251 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:01:36.252 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2024-05-09T05:01:36.252 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2024-05-09T05:01:36.252 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:01:36.252 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:36.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.252 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2024-05-09T05:01:36.252 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQA0WDxmrlOkJRAAf6eB9x9UcDCilfbWBIMCGQ==", 2024-05-09T05:01:36.252 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:01:36.252 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2024-05-09T05:01:36.252 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2024-05-09T05:01:36.252 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:01:36.252 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:36.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.253 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2024-05-09T05:01:36.253 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQA3WDxm2ItKJBAAO4FLP3sH/POba/k7DGFXqA==", 2024-05-09T05:01:36.253 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:01:36.253 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2024-05-09T05:01:36.253 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2024-05-09T05:01:36.253 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:01:36.253 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:36.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.253 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2024-05-09T05:01:36.253 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQA9WDxmKPPLKxAAGlyhDb222M4GpRybQ24Z3g==", 2024-05-09T05:01:36.254 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:01:36.254 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2024-05-09T05:01:36.254 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2024-05-09T05:01:36.254 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:01:36.254 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.254 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:36.254 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.254 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2024-05-09T05:01:36.254 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBAWDxmRKm6KRAAcZrQzTb6SQW5CQPyW9f+uQ==", 2024-05-09T05:01:36.254 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:01:36.254 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2024-05-09T05:01:36.254 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2024-05-09T05:01:36.254 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:01:36.255 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:36.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.255 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2024-05-09T05:01:36.255 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBDWDxmQkRwKBAA5/kHAwsRHh27fnYJuUh16A==", 2024-05-09T05:01:36.255 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:01:36.255 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2024-05-09T05:01:36.255 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2024-05-09T05:01:36.255 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:01:36.255 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:36.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.255 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2024-05-09T05:01:36.255 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQBGWDxmAc98JhAAA/qroXPpF1fC79NGd7WN4w==", 2024-05-09T05:01:36.255 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:01:36.256 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow profile osd", 2024-05-09T05:01:36.256 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile osd", 2024-05-09T05:01:36.256 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:01:36.256 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:36.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.256 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2024-05-09T05:01:36.256 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAtWDxmiEa1CBAAm5ehYW6MVg0B6WUq8TxNbg==", 2024-05-09T05:01:36.256 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:01:36.256 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2024-05-09T05:01:36.256 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2024-05-09T05:01:36.256 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-05-09T05:01:36.256 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:36.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.257 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2024-05-09T05:01:36.257 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAtWDxmA/enERAABGFsfdUaqliUPbTu5oCWJQ==", 2024-05-09T05:01:36.257 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:01:36.257 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2024-05-09T05:01:36.257 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2024-05-09T05:01:36.257 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2024-05-09T05:01:36.257 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2024-05-09T05:01:36.257 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:36.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.257 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2024-05-09T05:01:36.257 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAtWDxmtWd4DhAAF1LGmIJOK3X1UoC2pTnOLw==", 2024-05-09T05:01:36.257 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:01:36.258 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2024-05-09T05:01:36.258 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2024-05-09T05:01:36.258 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2024-05-09T05:01:36.258 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2024-05-09T05:01:36.258 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.258 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:36.258 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.258 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.3", 2024-05-09T05:01:36.258 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAtWDxmFWFIFxAAY/B8R3xXgu+A509Ww5Hqjg==", 2024-05-09T05:01:36.258 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:01:36.258 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2024-05-09T05:01:36.258 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2024-05-09T05:01:36.258 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2024-05-09T05:01:36.259 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2024-05-09T05:01:36.259 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:36.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.259 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2024-05-09T05:01:36.259 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAsWDxmxY1wERAAjAXnXGh6+bKKDWkUu6TWJw==", 2024-05-09T05:01:36.259 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:01:36.259 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2024-05-09T05:01:36.259 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2024-05-09T05:01:36.259 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2024-05-09T05:01:36.259 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:01:36.259 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:36.260 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.260 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2024-05-09T05:01:36.260 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAsWDxmuRPmKRAA8uT8sJ8VL1XVxa6c3RTj6g==", 2024-05-09T05:01:36.260 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:01:36.260 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2024-05-09T05:01:36.260 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2024-05-09T05:01:36.260 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:01:36.260 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-05-09T05:01:36.260 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.260 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2024-05-09T05:01:36.260 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQAsWDxmdBNeJhAAyU0fz/hMt2qp285unqOtlQ==", 2024-05-09T05:01:36.260 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2024-05-09T05:01:36.260 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2024-05-09T05:01:36.261 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2024-05-09T05:01:36.261 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2024-05-09T05:01:36.261 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.261 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.261 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-05-09T05:01:36.261 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:01:36.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.248+0000 7f52e2124640 1 -- 172.21.15.60:0/2745519400 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f52c80762d0 msgr2=0x7f52c8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:36.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.248+0000 7f52e2124640 1 --2- 172.21.15.60:0/2745519400 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f52c80762d0 0x7f52c8078790 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f52cc009b40 tx=0x7f52cc009190 comp rx=0 tx=0).stop 2024-05-09T05:01:36.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.248+0000 7f52e2124640 1 -- 172.21.15.60:0/2745519400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52dc09c8b0 msgr2=0x7f52dc15f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:36.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.248+0000 7f52e2124640 1 --2- 172.21.15.60:0/2745519400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52dc09c8b0 0x7f52dc15f940 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f52d000c450 tx=0x7f52d000c910 comp rx=0 tx=0).stop 2024-05-09T05:01:36.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.248+0000 7f52e2124640 1 -- 172.21.15.60:0/2745519400 shutdown_connections 2024-05-09T05:01:36.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.248+0000 7f52e2124640 1 --2- 172.21.15.60:0/2745519400 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f52c80762d0 0x7f52c8078790 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.248+0000 7f52e2124640 1 --2- 172.21.15.60:0/2745519400 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f52dc15f3e0 0x7f52dc0ac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.248+0000 7f52e2124640 1 --2- 172.21.15.60:0/2745519400 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52dc09c8b0 0x7f52dc15f940 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.248+0000 7f52e2124640 1 --2- 172.21.15.60:0/2745519400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52dc073310 0x7f52dc0d4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.248+0000 7f52e2124640 1 -- 172.21.15.60:0/2745519400 >> 172.21.15.60:0/2745519400 conn(0x7f52dc0d1470 msgr2=0x7f52dc15dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:36.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.248+0000 7f52e2124640 1 -- 172.21.15.60:0/2745519400 shutdown_connections 2024-05-09T05:01:36.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.248+0000 7f52e2124640 1 -- 172.21.15.60:0/2745519400 wait complete. 2024-05-09T05:01:36.264 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-09T05:01:36.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.368+0000 7fef85483640 1 Processor -- start 2024-05-09T05:01:36.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.368+0000 7fef85483640 1 -- start start 2024-05-09T05:01:36.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.368+0000 7fef85483640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef8015f300 0x7fef801639f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:36.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.368+0000 7fef85483640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fef8006f9f0 0x7fef800d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:36.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.368+0000 7fef85483640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fef800d3930 0x7fef8006ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:36.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.368+0000 7fef85483640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fef8006ff60 con 0x7fef8006f9f0 2024-05-09T05:01:36.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.368+0000 7fef85483640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fef80070240 con 0x7fef800d3930 2024-05-09T05:01:36.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.368+0000 7fef85483640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fef80070520 con 0x7fef8015f300 2024-05-09T05:01:36.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.369+0000 7fef7e7fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fef8006f9f0 0x7fef800d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:36.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.369+0000 7fef7e7fc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fef8006f9f0 0x7fef800d33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42040/0 (socket says 172.21.15.60:42040) 2024-05-09T05:01:36.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.369+0000 7fef7e7fc640 1 -- 172.21.15.60:0/2230911938 learned_addr learned my addr 172.21.15.60:0/2230911938 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:36.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.369+0000 7fef7effd640 1 --2- 172.21.15.60:0/2230911938 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef8015f300 0x7fef801639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:36.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.369+0000 7fef7f7fe640 1 --2- 172.21.15.60:0/2230911938 >> v2:172.21.15.114:3300/0 conn(0x7fef800d3930 0x7fef8006ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:36.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.369+0000 7fef7e7fc640 1 -- 172.21.15.60:0/2230911938 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef8015f300 msgr2=0x7fef801639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:36.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.369+0000 7fef7e7fc640 1 --2- 172.21.15.60:0/2230911938 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef8015f300 0x7fef801639f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.369+0000 7fef7e7fc640 1 -- 172.21.15.60:0/2230911938 >> v2:172.21.15.114:3300/0 conn(0x7fef800d3930 msgr2=0x7fef8006ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:36.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.369+0000 7fef7e7fc640 1 --2- 172.21.15.60:0/2230911938 >> v2:172.21.15.114:3300/0 conn(0x7fef800d3930 0x7fef8006ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.369+0000 7fef7e7fc640 1 -- 172.21.15.60:0/2230911938 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef8006e740 con 0x7fef8006f9f0 2024-05-09T05:01:36.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.369+0000 7fef7effd640 1 --2- 172.21.15.60:0/2230911938 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef8015f300 0x7fef801639f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:36.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.369+0000 7fef7f7fe640 1 --2- 172.21.15.60:0/2230911938 >> v2:172.21.15.114:3300/0 conn(0x7fef800d3930 0x7fef8006ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:36.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.370+0000 7fef7e7fc640 1 --2- 172.21.15.60:0/2230911938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef8006f9f0 0x7fef800d33f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fef8009f890 tx=0x7fef6802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=284824988f989fb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:36.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.370+0000 7fef7dffb640 1 -- 172.21.15.60:0/2230911938 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef68037070 con 0x7fef8006f9f0 2024-05-09T05:01:36.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.370+0000 7fef7dffb640 1 -- 172.21.15.60:0/2230911938 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef68032440 con 0x7fef8006f9f0 2024-05-09T05:01:36.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.370+0000 7fef7dffb640 1 -- 172.21.15.60:0/2230911938 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef68032810 con 0x7fef8006f9f0 2024-05-09T05:01:36.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.370+0000 7fef85483640 1 -- 172.21.15.60:0/2230911938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef8006f9f0 msgr2=0x7fef800d33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:36.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.370+0000 7fef85483640 1 --2- 172.21.15.60:0/2230911938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef8006f9f0 0x7fef800d33f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fef8009f890 tx=0x7fef6802a1e0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.370+0000 7fef85483640 1 -- 172.21.15.60:0/2230911938 shutdown_connections 2024-05-09T05:01:36.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.370+0000 7fef85483640 1 --2- 172.21.15.60:0/2230911938 >> v2:172.21.15.114:3300/0 conn(0x7fef800d3930 0x7fef8006ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.370+0000 7fef85483640 1 --2- 172.21.15.60:0/2230911938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef8006f9f0 0x7fef800d33f0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.370+0000 7fef85483640 1 --2- 172.21.15.60:0/2230911938 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef8015f300 0x7fef801639f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.370+0000 7fef85483640 1 -- 172.21.15.60:0/2230911938 >> 172.21.15.60:0/2230911938 conn(0x7fef8009b850 msgr2=0x7fef8009bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:36.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.370+0000 7fef85483640 1 -- 172.21.15.60:0/2230911938 shutdown_connections 2024-05-09T05:01:36.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.370+0000 7fef85483640 1 -- 172.21.15.60:0/2230911938 wait complete. 2024-05-09T05:01:36.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.372+0000 7fef85483640 1 Processor -- start 2024-05-09T05:01:36.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.372+0000 7fef85483640 1 -- start start 2024-05-09T05:01:36.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.372+0000 7fef85483640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef8006f9f0 0x7fef800aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:36.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.372+0000 7fef85483640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef800d3930 0x7fef800b2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:36.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.372+0000 7fef85483640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef8015f300 0x7fef800b2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:36.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.372+0000 7fef85483640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fef8006f480 con 0x7fef8015f300 2024-05-09T05:01:36.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.372+0000 7fef85483640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fef80163f30 con 0x7fef8006f9f0 2024-05-09T05:01:36.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.372+0000 7fef85483640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fef8006f600 con 0x7fef800d3930 2024-05-09T05:01:36.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.373+0000 7fef7f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef8015f300 0x7fef800b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:36.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.373+0000 7fef7f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef8015f300 0x7fef800b2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42048/0 (socket says 172.21.15.60:42048) 2024-05-09T05:01:36.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.373+0000 7fef7f7fe640 1 -- 172.21.15.60:0/2507410630 learned_addr learned my addr 172.21.15.60:0/2507410630 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:36.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.373+0000 7fef7e7fc640 1 --2- 172.21.15.60:0/2507410630 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef800d3930 0x7fef800b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:36.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.373+0000 7fef7effd640 1 --2- 172.21.15.60:0/2507410630 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef8006f9f0 0x7fef800aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:36.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.373+0000 7fef7f7fe640 1 -- 172.21.15.60:0/2507410630 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef800d3930 msgr2=0x7fef800b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:36.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.373+0000 7fef7f7fe640 1 --2- 172.21.15.60:0/2507410630 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef800d3930 0x7fef800b2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.373+0000 7fef7f7fe640 1 -- 172.21.15.60:0/2507410630 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef8006f9f0 msgr2=0x7fef800aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:36.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.373+0000 7fef7f7fe640 1 --2- 172.21.15.60:0/2507410630 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef8006f9f0 0x7fef800aff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.373+0000 7fef7f7fe640 1 -- 172.21.15.60:0/2507410630 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef801f57a0 con 0x7fef8015f300 2024-05-09T05:01:36.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.373+0000 7fef7f7fe640 1 --2- 172.21.15.60:0/2507410630 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef8015f300 0x7fef800b2840 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fef7800c7c0 tx=0x7fef7800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:36.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.373+0000 7fef777fe640 1 -- 172.21.15.60:0/2507410630 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef78015020 con 0x7fef8015f300 2024-05-09T05:01:36.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.373+0000 7fef777fe640 1 -- 172.21.15.60:0/2507410630 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef78004230 con 0x7fef8015f300 2024-05-09T05:01:36.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.373+0000 7fef85483640 1 -- 172.21.15.60:0/2507410630 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef801f5a80 con 0x7fef8015f300 2024-05-09T05:01:36.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.373+0000 7fef777fe640 1 -- 172.21.15.60:0/2507410630 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef78007620 con 0x7fef8015f300 2024-05-09T05:01:36.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.373+0000 7fef85483640 1 -- 172.21.15.60:0/2507410630 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef801f6040 con 0x7fef8015f300 2024-05-09T05:01:36.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.374+0000 7fef85483640 1 -- 172.21.15.60:0/2507410630 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef80051040 con 0x7fef8015f300 2024-05-09T05:01:36.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.375+0000 7fef777fe640 1 -- 172.21.15.60:0/2507410630 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fef78018020 con 0x7fef8015f300 2024-05-09T05:01:36.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.376+0000 7fef777fe640 1 --2- 172.21.15.60:0/2507410630 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fef600762d0 0x7fef60078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:36.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.376+0000 7fef777fe640 1 -- 172.21.15.60:0/2507410630 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5249+0+0 (secure 0 0 0) 0x7fef780972e0 con 0x7fef8015f300 2024-05-09T05:01:36.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.376+0000 7fef7effd640 1 --2- 172.21.15.60:0/2507410630 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fef600762d0 0x7fef60078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:36.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.376+0000 7fef7effd640 1 --2- 172.21.15.60:0/2507410630 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fef600762d0 0x7fef60078790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fef7000d540 tx=0x7fef70007810 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:36.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.380+0000 7fef777fe640 1 -- 172.21.15.60:0/2507410630 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fef7801f020 con 0x7fef8015f300 2024-05-09T05:01:36.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.573+0000 7fef85483640 1 -- 172.21.15.60:0/2507410630 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fef8009f8e0 con 0x7fef8015f300 2024-05-09T05:01:36.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.574+0000 7fef777fe640 1 -- 172.21.15.60:0/2507410630 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v47) ==== 84+0+65 (secure 0 0 0) 0x7fef78060900 con 0x7fef8015f300 2024-05-09T05:01:36.574 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:36.574 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2024-05-09T05:01:36.575 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-05-09T05:01:36.575 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2024-05-09T05:01:36.575 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2024-05-09T05:01:36.575 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-05-09T05:01:36.575 INFO:teuthology.orchestra.run.smithi060.stdout:] 2024-05-09T05:01:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.576+0000 7fef85483640 1 -- 172.21.15.60:0/2507410630 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fef600762d0 msgr2=0x7fef60078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.576+0000 7fef85483640 1 --2- 172.21.15.60:0/2507410630 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fef600762d0 0x7fef60078790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fef7000d540 tx=0x7fef70007810 comp rx=0 tx=0).stop 2024-05-09T05:01:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.576+0000 7fef85483640 1 -- 172.21.15.60:0/2507410630 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef8015f300 msgr2=0x7fef800b2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.576+0000 7fef85483640 1 --2- 172.21.15.60:0/2507410630 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef8015f300 0x7fef800b2840 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fef7800c7c0 tx=0x7fef7800cc80 comp rx=0 tx=0).stop 2024-05-09T05:01:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.576+0000 7fef85483640 1 -- 172.21.15.60:0/2507410630 shutdown_connections 2024-05-09T05:01:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.576+0000 7fef85483640 1 --2- 172.21.15.60:0/2507410630 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fef600762d0 0x7fef60078790 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.576+0000 7fef85483640 1 --2- 172.21.15.60:0/2507410630 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef8015f300 0x7fef800b2840 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.576+0000 7fef85483640 1 --2- 172.21.15.60:0/2507410630 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fef800d3930 0x7fef800b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.576+0000 7fef85483640 1 --2- 172.21.15.60:0/2507410630 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef8006f9f0 0x7fef800aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.576+0000 7fef85483640 1 -- 172.21.15.60:0/2507410630 >> 172.21.15.60:0/2507410630 conn(0x7fef8009b850 msgr2=0x7fef8009d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.576+0000 7fef85483640 1 -- 172.21.15.60:0/2507410630 shutdown_connections 2024-05-09T05:01:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.576+0000 7fef85483640 1 -- 172.21.15.60:0/2507410630 wait complete. 2024-05-09T05:01:36.587 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-09T05:01:36.587 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-09T05:01:36.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.696+0000 7f871dc2a640 1 Processor -- start 2024-05-09T05:01:36.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.696+0000 7f871dc2a640 1 -- start start 2024-05-09T05:01:36.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.696+0000 7f871dc2a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8718148120 0x7f87180d2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:36.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.696+0000 7f871dc2a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87180d2ca0 0x7f8718162e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:36.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.696+0000 7f871dc2a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f87181633b0 0x7f87180d4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:36.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.696+0000 7f871dc2a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f8718163800 con 0x7f87181633b0 2024-05-09T05:01:36.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.696+0000 7f871dc2a640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f87180d31b0 con 0x7f8718148120 2024-05-09T05:01:36.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.696+0000 7f871dc2a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f87180d3490 con 0x7f87180d2ca0 2024-05-09T05:01:36.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.697+0000 7f8717fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f87181633b0 0x7f87180d4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:36.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.697+0000 7f8717fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f87181633b0 0x7f87180d4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42054/0 (socket says 172.21.15.60:42054) 2024-05-09T05:01:36.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.697+0000 7f8717fff640 1 -- 172.21.15.60:0/27939587 learned_addr learned my addr 172.21.15.60:0/27939587 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:36.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.697+0000 7f8716ffd640 1 --2- 172.21.15.60:0/27939587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87180d2ca0 0x7f8718162e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:36.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.697+0000 7f8717fff640 1 -- 172.21.15.60:0/27939587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87180d2ca0 msgr2=0x7f8718162e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:36.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.697+0000 7f87177fe640 1 --2- 172.21.15.60:0/27939587 >> v2:172.21.15.114:3300/0 conn(0x7f8718148120 0x7f87180d2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:36.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.697+0000 7f8717fff640 1 --2- 172.21.15.60:0/27939587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87180d2ca0 0x7f8718162e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.697+0000 7f8717fff640 1 -- 172.21.15.60:0/27939587 >> v2:172.21.15.114:3300/0 conn(0x7f8718148120 msgr2=0x7f87180d2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:36.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.697+0000 7f8717fff640 1 --2- 172.21.15.60:0/27939587 >> v2:172.21.15.114:3300/0 conn(0x7f8718148120 0x7f87180d2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.697+0000 7f8717fff640 1 -- 172.21.15.60:0/27939587 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87180b5650 con 0x7f87181633b0 2024-05-09T05:01:36.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.697+0000 7f8716ffd640 1 --2- 172.21.15.60:0/27939587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87180d2ca0 0x7f8718162e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:36.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.697+0000 7f87177fe640 1 --2- 172.21.15.60:0/27939587 >> v2:172.21.15.114:3300/0 conn(0x7f8718148120 0x7f87180d2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:36.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.698+0000 7f8717fff640 1 --2- 172.21.15.60:0/27939587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87181633b0 0x7f87180d4020 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8718075560 tx=0x7f871002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a609d9e50966a7f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:36.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.698+0000 7f87167fc640 1 -- 172.21.15.60:0/27939587 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8710036070 con 0x7f87181633b0 2024-05-09T05:01:36.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.698+0000 7f87167fc640 1 -- 172.21.15.60:0/27939587 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8710031600 con 0x7f87181633b0 2024-05-09T05:01:36.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.698+0000 7f87167fc640 1 -- 172.21.15.60:0/27939587 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87100319d0 con 0x7f87181633b0 2024-05-09T05:01:36.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.698+0000 7f871dc2a640 1 -- 172.21.15.60:0/27939587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87181633b0 msgr2=0x7f87180d4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:36.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.698+0000 7f871dc2a640 1 --2- 172.21.15.60:0/27939587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87181633b0 0x7f87180d4020 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8718075560 tx=0x7f871002a6a0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.698+0000 7f871dc2a640 1 -- 172.21.15.60:0/27939587 shutdown_connections 2024-05-09T05:01:36.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.698+0000 7f871dc2a640 1 --2- 172.21.15.60:0/27939587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87181633b0 0x7f87180d4020 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.698+0000 7f871dc2a640 1 --2- 172.21.15.60:0/27939587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f87180d2ca0 0x7f8718162e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.698+0000 7f871dc2a640 1 --2- 172.21.15.60:0/27939587 >> v2:172.21.15.114:3300/0 conn(0x7f8718148120 0x7f87180d2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.698+0000 7f871dc2a640 1 -- 172.21.15.60:0/27939587 >> 172.21.15.60:0/27939587 conn(0x7f871806f640 msgr2=0x7f8718073840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:36.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.698+0000 7f871dc2a640 1 -- 172.21.15.60:0/27939587 shutdown_connections 2024-05-09T05:01:36.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.698+0000 7f871dc2a640 1 -- 172.21.15.60:0/27939587 wait complete. 2024-05-09T05:01:36.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.700+0000 7f871dc2a640 1 Processor -- start 2024-05-09T05:01:36.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.700+0000 7f871dc2a640 1 -- start start 2024-05-09T05:01:36.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f871dc2a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87180d2ca0 0x7f8718073710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:36.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f871dc2a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8718148120 0x7f8718161840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:36.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f871dc2a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87181633b0 0x7f871813c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:36.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f871dc2a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8718163b30 con 0x7f87180d2ca0 2024-05-09T05:01:36.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f871dc2a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8718148500 con 0x7f87181633b0 2024-05-09T05:01:36.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f871dc2a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8718163cb0 con 0x7f8718148120 2024-05-09T05:01:36.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f87177fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87180d2ca0 0x7f8718073710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:36.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f87177fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87180d2ca0 0x7f8718073710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42070/0 (socket says 172.21.15.60:42070) 2024-05-09T05:01:36.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f87177fe640 1 -- 172.21.15.60:0/1748614278 learned_addr learned my addr 172.21.15.60:0/1748614278 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:36.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f8716ffd640 1 --2- 172.21.15.60:0/1748614278 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8718148120 0x7f8718161840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:36.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f8717fff640 1 --2- 172.21.15.60:0/1748614278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87181633b0 0x7f871813c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:36.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f87177fe640 1 -- 172.21.15.60:0/1748614278 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8718148120 msgr2=0x7f8718161840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:36.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f87177fe640 1 --2- 172.21.15.60:0/1748614278 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8718148120 0x7f8718161840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f87177fe640 1 -- 172.21.15.60:0/1748614278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87181633b0 msgr2=0x7f871813c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:36.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f87177fe640 1 --2- 172.21.15.60:0/1748614278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87181633b0 0x7f871813c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:36.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f87177fe640 1 -- 172.21.15.60:0/1748614278 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8718072670 con 0x7f87180d2ca0 2024-05-09T05:01:36.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f8717fff640 1 --2- 172.21.15.60:0/1748614278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87181633b0 0x7f871813c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:36.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f87177fe640 1 --2- 172.21.15.60:0/1748614278 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87180d2ca0 0x7f8718073710 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f870800c820 tx=0x7f870800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:36.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f86fffff640 1 -- 172.21.15.60:0/1748614278 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8708015070 con 0x7f87180d2ca0 2024-05-09T05:01:36.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f86fffff640 1 -- 172.21.15.60:0/1748614278 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87080079d0 con 0x7f87180d2ca0 2024-05-09T05:01:36.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f871dc2a640 1 -- 172.21.15.60:0/1748614278 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8718072950 con 0x7f87180d2ca0 2024-05-09T05:01:36.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f86fffff640 1 -- 172.21.15.60:0/1748614278 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8708007da0 con 0x7f87180d2ca0 2024-05-09T05:01:36.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.701+0000 7f871dc2a640 1 -- 172.21.15.60:0/1748614278 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8718072f10 con 0x7f87180d2ca0 2024-05-09T05:01:36.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.702+0000 7f871dc2a640 1 -- 172.21.15.60:0/1748614278 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8718051040 con 0x7f87180d2ca0 2024-05-09T05:01:36.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.703+0000 7f86fffff640 1 -- 172.21.15.60:0/1748614278 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8708018070 con 0x7f87180d2ca0 2024-05-09T05:01:36.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.703+0000 7f86fffff640 1 --2- 172.21.15.60:0/1748614278 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f87040762d0 0x7f8704078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:36.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.704+0000 7f86fffff640 1 -- 172.21.15.60:0/1748614278 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5249+0+0 (secure 0 0 0) 0x7f8708097260 con 0x7f87180d2ca0 2024-05-09T05:01:36.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.704+0000 7f8716ffd640 1 --2- 172.21.15.60:0/1748614278 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f87040762d0 0x7f8704078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:36.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.704+0000 7f8716ffd640 1 --2- 172.21.15.60:0/1748614278 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f87040762d0 0x7f8704078790 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8718140800 tx=0x7f870c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:36.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.708+0000 7f86fffff640 1 -- 172.21.15.60:0/1748614278 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f87080607c0 con 0x7f87180d2ca0 2024-05-09T05:01:36.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:36.905+0000 7f871dc2a640 1 -- 172.21.15.60:0/1748614278 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f8718162be0 con 0x7f87180d2ca0 2024-05-09T05:01:37.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.839+0000 7f86fffff640 1 -- 172.21.15.60:0/1748614278 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v48) ==== 138+0+0 (secure 0 0 0) 0x7f8708064ac0 con 0x7f87180d2ca0 2024-05-09T05:01:37.840 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2024-05-09T05:01:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.842+0000 7f871dc2a640 1 -- 172.21.15.60:0/1748614278 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f87040762d0 msgr2=0x7f8704078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.842+0000 7f871dc2a640 1 --2- 172.21.15.60:0/1748614278 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f87040762d0 0x7f8704078790 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8718140800 tx=0x7f870c008040 comp rx=0 tx=0).stop 2024-05-09T05:01:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.842+0000 7f871dc2a640 1 -- 172.21.15.60:0/1748614278 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87180d2ca0 msgr2=0x7f8718073710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.842+0000 7f871dc2a640 1 --2- 172.21.15.60:0/1748614278 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87180d2ca0 0x7f8718073710 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f870800c820 tx=0x7f870800cc20 comp rx=0 tx=0).stop 2024-05-09T05:01:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.842+0000 7f871dc2a640 1 -- 172.21.15.60:0/1748614278 shutdown_connections 2024-05-09T05:01:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.842+0000 7f871dc2a640 1 --2- 172.21.15.60:0/1748614278 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f87040762d0 0x7f8704078790 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:37.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.842+0000 7f871dc2a640 1 --2- 172.21.15.60:0/1748614278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87181633b0 0x7f871813c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:37.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.842+0000 7f871dc2a640 1 --2- 172.21.15.60:0/1748614278 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8718148120 0x7f8718161840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:37.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.842+0000 7f871dc2a640 1 --2- 172.21.15.60:0/1748614278 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87180d2ca0 0x7f8718073710 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:37.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.842+0000 7f871dc2a640 1 -- 172.21.15.60:0/1748614278 >> 172.21.15.60:0/1748614278 conn(0x7f871806f640 msgr2=0x7f8718144320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:37.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.843+0000 7f871dc2a640 1 -- 172.21.15.60:0/1748614278 shutdown_connections 2024-05-09T05:01:37.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.843+0000 7f871dc2a640 1 -- 172.21.15.60:0/1748614278 wait complete. 2024-05-09T05:01:37.861 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-09T05:01:37.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.977+0000 7fb5172d1640 1 Processor -- start 2024-05-09T05:01:37.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.977+0000 7fb5172d1640 1 -- start start 2024-05-09T05:01:37.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.978+0000 7fb5172d1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb51015f330 0x7fb5100b7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:37.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.978+0000 7fb5172d1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb5100b65d0 0x7fb5100730e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:37.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.978+0000 7fb5172d1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb510072c00 con 0x7fb510162fa0 2024-05-09T05:01:37.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.978+0000 7fb5172d1640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fb510163540 con 0x7fb5100b65d0 2024-05-09T05:01:37.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.978+0000 7fb5172d1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb510163820 con 0x7fb51015f330 2024-05-09T05:01:37.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.978+0000 7fb51586e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb5100b65d0 0x7fb5100730e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:37.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.978+0000 7fb51586e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb5100b65d0 0x7fb5100730e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42076/0 (socket says 172.21.15.60:42076) 2024-05-09T05:01:37.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.978+0000 7fb51586e640 1 -- 172.21.15.60:0/3266548648 learned_addr learned my addr 172.21.15.60:0/3266548648 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:37.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.978+0000 7fb51506d640 1 --2- 172.21.15.60:0/3266548648 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb51015f330 0x7fb5100b7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:37.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.979+0000 7fb51586e640 1 -- 172.21.15.60:0/3266548648 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb51015f330 msgr2=0x7fb5100b7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:37.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.979+0000 7fb51586e640 1 --2- 172.21.15.60:0/3266548648 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb51015f330 0x7fb5100b7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:37.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.979+0000 7fb51586e640 1 -- 172.21.15.60:0/3266548648 >> v1:172.21.15.114:6789/0 conn(0x7fb510162fa0 legacy=0x7fb5100b5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:37.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.979+0000 7fb51586e640 1 -- 172.21.15.60:0/3266548648 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb510163b00 con 0x7fb5100b65d0 2024-05-09T05:01:37.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.979+0000 7fb51506d640 1 --2- 172.21.15.60:0/3266548648 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb51015f330 0x7fb5100b7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:37.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.979+0000 7fb51586e640 1 --2- 172.21.15.60:0/3266548648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5100b65d0 0x7fb5100730e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb500009e50 tx=0x7fb50002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d78d11117ea2c82 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:37.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.979+0000 7fb50ffff640 1 -- 172.21.15.60:0/3266548648 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb500036070 con 0x7fb5100b65d0 2024-05-09T05:01:37.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.980+0000 7fb50ffff640 1 -- 172.21.15.60:0/3266548648 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb500031600 con 0x7fb5100b65d0 2024-05-09T05:01:37.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.980+0000 7fb50ffff640 1 -- 172.21.15.60:0/3266548648 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5000319d0 con 0x7fb5100b65d0 2024-05-09T05:01:37.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.980+0000 7fb5172d1640 1 -- 172.21.15.60:0/3266548648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5100b65d0 msgr2=0x7fb5100730e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:37.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.980+0000 7fb5172d1640 1 --2- 172.21.15.60:0/3266548648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5100b65d0 0x7fb5100730e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb500009e50 tx=0x7fb50002a680 comp rx=0 tx=0).stop 2024-05-09T05:01:37.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.980+0000 7fb5172d1640 1 -- 172.21.15.60:0/3266548648 shutdown_connections 2024-05-09T05:01:37.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.980+0000 7fb5172d1640 1 --2- 172.21.15.60:0/3266548648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5100b65d0 0x7fb5100730e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:37.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.980+0000 7fb5172d1640 1 --2- 172.21.15.60:0/3266548648 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb51015f330 0x7fb5100b7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:37.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.980+0000 7fb5172d1640 1 -- 172.21.15.60:0/3266548648 >> 172.21.15.60:0/3266548648 conn(0x7fb51006ca80 msgr2=0x7fb5100d53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:37.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.980+0000 7fb5172d1640 1 -- 172.21.15.60:0/3266548648 shutdown_connections 2024-05-09T05:01:37.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.980+0000 7fb5172d1640 1 -- 172.21.15.60:0/3266548648 wait complete. 2024-05-09T05:01:37.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.981+0000 7fb5172d1640 1 Processor -- start 2024-05-09T05:01:37.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.981+0000 7fb5172d1640 1 -- start start 2024-05-09T05:01:37.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.982+0000 7fb5172d1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5100b65d0 0x7fb510156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:37.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.982+0000 7fb5172d1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb51015f330 0x7fb51015d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:37.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.982+0000 7fb5172d1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb510162fa0 0x7fb510140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:37.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.982+0000 7fb5172d1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb51015f710 con 0x7fb51015f330 2024-05-09T05:01:37.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.982+0000 7fb5172d1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb510163de0 con 0x7fb5100b65d0 2024-05-09T05:01:37.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.982+0000 7fb5172d1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb51015e070 con 0x7fb510162fa0 2024-05-09T05:01:37.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.982+0000 7fb51486c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb51015f330 0x7fb51015d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:37.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.982+0000 7fb51486c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb51015f330 0x7fb51015d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42088/0 (socket says 172.21.15.60:42088) 2024-05-09T05:01:37.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.982+0000 7fb51486c640 1 -- 172.21.15.60:0/700362359 learned_addr learned my addr 172.21.15.60:0/700362359 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:37.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.982+0000 7fb51506d640 1 --2- 172.21.15.60:0/700362359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5100b65d0 0x7fb510156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:37.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.982+0000 7fb51586e640 1 --2- 172.21.15.60:0/700362359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb510162fa0 0x7fb510140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:37.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.983+0000 7fb51486c640 1 -- 172.21.15.60:0/700362359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb510162fa0 msgr2=0x7fb510140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:37.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.983+0000 7fb51486c640 1 --2- 172.21.15.60:0/700362359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb510162fa0 0x7fb510140ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:37.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.983+0000 7fb51486c640 1 -- 172.21.15.60:0/700362359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5100b65d0 msgr2=0x7fb510156570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:37.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.983+0000 7fb51486c640 1 --2- 172.21.15.60:0/700362359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5100b65d0 0x7fb510156570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:37.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.983+0000 7fb51486c640 1 -- 172.21.15.60:0/700362359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb51014bbb0 con 0x7fb51015f330 2024-05-09T05:01:37.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.983+0000 7fb51486c640 1 --2- 172.21.15.60:0/700362359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb51015f330 0x7fb51015d820 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb50400b3f0 tx=0x7fb50400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:37.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.983+0000 7fb50dffb640 1 -- 172.21.15.60:0/700362359 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb504015020 con 0x7fb51015f330 2024-05-09T05:01:37.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.983+0000 7fb50dffb640 1 -- 172.21.15.60:0/700362359 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb504011830 con 0x7fb51015f330 2024-05-09T05:01:37.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.983+0000 7fb5172d1640 1 -- 172.21.15.60:0/700362359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb51014be90 con 0x7fb51015f330 2024-05-09T05:01:37.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.983+0000 7fb50dffb640 1 -- 172.21.15.60:0/700362359 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb504011c20 con 0x7fb51015f330 2024-05-09T05:01:37.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.983+0000 7fb5172d1640 1 -- 172.21.15.60:0/700362359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb51015dd60 con 0x7fb51015f330 2024-05-09T05:01:37.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.984+0000 7fb5172d1640 1 -- 172.21.15.60:0/700362359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb510051040 con 0x7fb51015f330 2024-05-09T05:01:37.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.985+0000 7fb50dffb640 1 -- 172.21.15.60:0/700362359 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb504018020 con 0x7fb51015f330 2024-05-09T05:01:37.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.985+0000 7fb50dffb640 1 --2- 172.21.15.60:0/700362359 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb4fc0762d0 0x7fb4fc078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:37.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.986+0000 7fb50dffb640 1 -- 172.21.15.60:0/700362359 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5619+0+0 (secure 0 0 0) 0x7fb504097310 con 0x7fb51015f330 2024-05-09T05:01:37.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.986+0000 7fb51506d640 1 --2- 172.21.15.60:0/700362359 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb4fc0762d0 0x7fb4fc078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:37.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.986+0000 7fb51506d640 1 --2- 172.21.15.60:0/700362359 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb4fc0762d0 0x7fb4fc078790 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb50800a990 tx=0x7fb508008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:37.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:37.990+0000 7fb50dffb640 1 -- 172.21.15.60:0/700362359 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb50401f030 con 0x7fb51015f330 2024-05-09T05:01:38.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.180+0000 7fb5172d1640 1 -- 172.21.15.60:0/700362359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb5100ba950 con 0x7fb51015f330 2024-05-09T05:01:38.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.845+0000 7fb50dffb640 1 -- 172.21.15.60:0/700362359 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v49) ==== 170+0+0 (secure 0 0 0) 0x7fb504060b90 con 0x7fb51015f330 2024-05-09T05:01:38.845 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2024-05-09T05:01:38.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.849+0000 7fb5172d1640 1 -- 172.21.15.60:0/700362359 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb4fc0762d0 msgr2=0x7fb4fc078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:38.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.850+0000 7fb5172d1640 1 --2- 172.21.15.60:0/700362359 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb4fc0762d0 0x7fb4fc078790 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb50800a990 tx=0x7fb508008040 comp rx=0 tx=0).stop 2024-05-09T05:01:38.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.850+0000 7fb5172d1640 1 -- 172.21.15.60:0/700362359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb51015f330 msgr2=0x7fb51015d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:38.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.850+0000 7fb5172d1640 1 --2- 172.21.15.60:0/700362359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb51015f330 0x7fb51015d820 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb50400b3f0 tx=0x7fb50400b8b0 comp rx=0 tx=0).stop 2024-05-09T05:01:38.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.856+0000 7fb5172d1640 1 -- 172.21.15.60:0/700362359 shutdown_connections 2024-05-09T05:01:38.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.856+0000 7fb5172d1640 1 --2- 172.21.15.60:0/700362359 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb4fc0762d0 0x7fb4fc078790 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:38.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.856+0000 7fb5172d1640 1 --2- 172.21.15.60:0/700362359 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb510162fa0 0x7fb510140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:38.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.856+0000 7fb5172d1640 1 --2- 172.21.15.60:0/700362359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb51015f330 0x7fb51015d820 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:38.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.856+0000 7fb5172d1640 1 --2- 172.21.15.60:0/700362359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5100b65d0 0x7fb510156570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:38.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.856+0000 7fb5172d1640 1 -- 172.21.15.60:0/700362359 >> 172.21.15.60:0/700362359 conn(0x7fb51006ca80 msgr2=0x7fb5100b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:38.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.860+0000 7fb5172d1640 1 -- 172.21.15.60:0/700362359 shutdown_connections 2024-05-09T05:01:38.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.862+0000 7fb5172d1640 1 -- 172.21.15.60:0/700362359 wait complete. 2024-05-09T05:01:38.879 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-09T05:01:38.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.991+0000 7ff414dc9640 1 Processor -- start 2024-05-09T05:01:38.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.991+0000 7ff414dc9640 1 -- start start 2024-05-09T05:01:38.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.992+0000 7ff414dc9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff4100b7fe0 0x7ff4100b95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:38.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.992+0000 7ff414dc9640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4100b9b30 0x7ff4100732d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:38.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.992+0000 7ff414dc9640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff410163740 con 0x7ff41015f3c0 2024-05-09T05:01:38.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.992+0000 7ff414dc9640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7ff410163a20 con 0x7ff4100b7fe0 2024-05-09T05:01:38.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.992+0000 7ff414dc9640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff410163d00 con 0x7ff4100b9b30 2024-05-09T05:01:38.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.992+0000 7ff407fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff4100b7fe0 0x7ff4100b95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:38.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.992+0000 7ff407fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff4100b7fe0 0x7ff4100b95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37582/0 (socket says 172.21.15.60:37582) 2024-05-09T05:01:38.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.992+0000 7ff407fff640 1 -- 172.21.15.60:0/3174406684 learned_addr learned my addr 172.21.15.60:0/3174406684 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:38.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.992+0000 7ff40ed9d640 1 --2- 172.21.15.60:0/3174406684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4100b9b30 0x7ff4100732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:38.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.992+0000 7ff407fff640 1 -- 172.21.15.60:0/3174406684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4100b9b30 msgr2=0x7ff4100732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:38.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.992+0000 7ff407fff640 1 --2- 172.21.15.60:0/3174406684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4100b9b30 0x7ff4100732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:38.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.992+0000 7ff407fff640 1 -- 172.21.15.60:0/3174406684 >> v1:172.21.15.114:6789/0 conn(0x7ff41015f3c0 legacy=0x7ff410162e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:38.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.992+0000 7ff407fff640 1 -- 172.21.15.60:0/3174406684 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff410072ad0 con 0x7ff4100b7fe0 2024-05-09T05:01:38.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.993+0000 7ff407fff640 1 --2- 172.21.15.60:0/3174406684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4100b7fe0 0x7ff4100b95f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8009870 tx=0x7ff3f802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d250b06395994f7b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:38.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.993+0000 7ff40dd9b640 1 -- 172.21.15.60:0/3174406684 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8037070 con 0x7ff4100b7fe0 2024-05-09T05:01:38.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.993+0000 7ff40dd9b640 1 -- 172.21.15.60:0/3174406684 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3f8032440 con 0x7ff4100b7fe0 2024-05-09T05:01:38.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.993+0000 7ff40dd9b640 1 -- 172.21.15.60:0/3174406684 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8032810 con 0x7ff4100b7fe0 2024-05-09T05:01:38.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.994+0000 7ff414dc9640 1 -- 172.21.15.60:0/3174406684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4100b7fe0 msgr2=0x7ff4100b95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:38.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.994+0000 7ff414dc9640 1 --2- 172.21.15.60:0/3174406684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4100b7fe0 0x7ff4100b95f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8009870 tx=0x7ff3f802a270 comp rx=0 tx=0).stop 2024-05-09T05:01:38.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.994+0000 7ff414dc9640 1 -- 172.21.15.60:0/3174406684 shutdown_connections 2024-05-09T05:01:38.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.994+0000 7ff414dc9640 1 --2- 172.21.15.60:0/3174406684 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4100b9b30 0x7ff4100732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:38.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.994+0000 7ff414dc9640 1 --2- 172.21.15.60:0/3174406684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4100b7fe0 0x7ff4100b95f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:38.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.994+0000 7ff414dc9640 1 -- 172.21.15.60:0/3174406684 >> 172.21.15.60:0/3174406684 conn(0x7ff41006cae0 msgr2=0x7ff4100d53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:38.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.994+0000 7ff414dc9640 1 -- 172.21.15.60:0/3174406684 shutdown_connections 2024-05-09T05:01:38.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.994+0000 7ff414dc9640 1 -- 172.21.15.60:0/3174406684 wait complete. 2024-05-09T05:01:38.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.995+0000 7ff414dc9640 1 Processor -- start 2024-05-09T05:01:38.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.995+0000 7ff414dc9640 1 -- start start 2024-05-09T05:01:38.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.996+0000 7ff414dc9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4100b7fe0 0x7ff410156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:38.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.996+0000 7ff414dc9640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4100b9b30 0x7ff410155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:38.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.996+0000 7ff414dc9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff41015f3c0 0x7ff41006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:38.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.996+0000 7ff414dc9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff410073810 con 0x7ff41015f3c0 2024-05-09T05:01:38.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.996+0000 7ff414dc9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff4100d1dc0 con 0x7ff4100b7fe0 2024-05-09T05:01:38.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.996+0000 7ff414dc9640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff410073990 con 0x7ff4100b9b30 2024-05-09T05:01:38.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.996+0000 7ff40ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff41015f3c0 0x7ff41006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:38.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.996+0000 7ff40ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff41015f3c0 0x7ff41006ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37596/0 (socket says 172.21.15.60:37596) 2024-05-09T05:01:38.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.996+0000 7ff40ed9d640 1 -- 172.21.15.60:0/2570633389 learned_addr learned my addr 172.21.15.60:0/2570633389 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:38.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.996+0000 7ff407fff640 1 --2- 172.21.15.60:0/2570633389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4100b9b30 0x7ff410155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:38.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.996+0000 7ff40e59c640 1 --2- 172.21.15.60:0/2570633389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4100b7fe0 0x7ff410156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:38.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.996+0000 7ff40ed9d640 1 -- 172.21.15.60:0/2570633389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4100b9b30 msgr2=0x7ff410155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:38.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.996+0000 7ff40ed9d640 1 --2- 172.21.15.60:0/2570633389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4100b9b30 0x7ff410155bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:38.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.996+0000 7ff40ed9d640 1 -- 172.21.15.60:0/2570633389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4100b7fe0 msgr2=0x7ff410156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:38.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.996+0000 7ff40ed9d640 1 --2- 172.21.15.60:0/2570633389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4100b7fe0 0x7ff410156d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:38.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.996+0000 7ff40ed9d640 1 -- 172.21.15.60:0/2570633389 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff410156100 con 0x7ff41015f3c0 2024-05-09T05:01:38.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.996+0000 7ff40ed9d640 1 --2- 172.21.15.60:0/2570633389 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff41015f3c0 0x7ff41006ed30 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff408007e40 tx=0x7ff40800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:38.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.997+0000 7ff406ffd640 1 -- 172.21.15.60:0/2570633389 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff408014620 con 0x7ff41015f3c0 2024-05-09T05:01:39.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.997+0000 7ff406ffd640 1 -- 172.21.15.60:0/2570633389 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff408019070 con 0x7ff41015f3c0 2024-05-09T05:01:39.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.997+0000 7ff414dc9640 1 -- 172.21.15.60:0/2570633389 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4101e8c50 con 0x7ff41015f3c0 2024-05-09T05:01:39.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.997+0000 7ff406ffd640 1 -- 172.21.15.60:0/2570633389 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff408014a20 con 0x7ff41015f3c0 2024-05-09T05:01:39.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.997+0000 7ff414dc9640 1 -- 172.21.15.60:0/2570633389 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4101e9210 con 0x7ff41015f3c0 2024-05-09T05:01:39.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.998+0000 7ff414dc9640 1 -- 172.21.15.60:0/2570633389 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff410051040 con 0x7ff41015f3c0 2024-05-09T05:01:39.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.998+0000 7ff406ffd640 1 -- 172.21.15.60:0/2570633389 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff408014bc0 con 0x7ff41015f3c0 2024-05-09T05:01:39.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.999+0000 7ff406ffd640 1 --2- 172.21.15.60:0/2570633389 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff3f00762d0 0x7ff3f0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:39.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.999+0000 7ff406ffd640 1 -- 172.21.15.60:0/2570633389 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6001+0+0 (secure 0 0 0) 0x7ff408098de0 con 0x7ff41015f3c0 2024-05-09T05:01:39.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:38.999+0000 7ff40e59c640 1 --2- 172.21.15.60:0/2570633389 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff3f00762d0 0x7ff3f0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:39.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.000+0000 7ff40e59c640 1 --2- 172.21.15.60:0/2570633389 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff3f00762d0 0x7ff3f0078790 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff4000015a0 tx=0x7ff400009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:39.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.003+0000 7ff406ffd640 1 -- 172.21.15.60:0/2570633389 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff4080626f0 con 0x7ff41015f3c0 2024-05-09T05:01:39.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.236+0000 7ff414dc9640 1 -- 172.21.15.60:0/2570633389 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ff4100ba900 con 0x7ff41015f3c0 2024-05-09T05:01:39.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.846+0000 7ff406ffd640 1 -- 172.21.15.60:0/2570633389 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v23) ==== 172+0+0 (secure 0 0 0) 0x7ff408061ff0 con 0x7ff41015f3c0 2024-05-09T05:01:39.847 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 8 and data pool 9 2024-05-09T05:01:39.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.851+0000 7ff414dc9640 1 -- 172.21.15.60:0/2570633389 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff3f00762d0 msgr2=0x7ff3f0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:39.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.851+0000 7ff414dc9640 1 --2- 172.21.15.60:0/2570633389 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff3f00762d0 0x7ff3f0078790 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff4000015a0 tx=0x7ff400009040 comp rx=0 tx=0).stop 2024-05-09T05:01:39.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.851+0000 7ff414dc9640 1 -- 172.21.15.60:0/2570633389 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff41015f3c0 msgr2=0x7ff41006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:39.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.851+0000 7ff414dc9640 1 --2- 172.21.15.60:0/2570633389 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff41015f3c0 0x7ff41006ed30 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff408007e40 tx=0x7ff40800e4b0 comp rx=0 tx=0).stop 2024-05-09T05:01:39.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.851+0000 7ff414dc9640 1 -- 172.21.15.60:0/2570633389 shutdown_connections 2024-05-09T05:01:39.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.851+0000 7ff414dc9640 1 --2- 172.21.15.60:0/2570633389 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff3f00762d0 0x7ff3f0078790 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:39.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.851+0000 7ff414dc9640 1 --2- 172.21.15.60:0/2570633389 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff41015f3c0 0x7ff41006ed30 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:39.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.851+0000 7ff414dc9640 1 --2- 172.21.15.60:0/2570633389 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff4100b9b30 0x7ff410155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:39.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.851+0000 7ff414dc9640 1 --2- 172.21.15.60:0/2570633389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4100b7fe0 0x7ff410156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:39.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.851+0000 7ff414dc9640 1 -- 172.21.15.60:0/2570633389 >> 172.21.15.60:0/2570633389 conn(0x7ff41006cae0 msgr2=0x7ff4100d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:39.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.852+0000 7ff414dc9640 1 -- 172.21.15.60:0/2570633389 shutdown_connections 2024-05-09T05:01:39.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.852+0000 7ff414dc9640 1 -- 172.21.15.60:0/2570633389 wait complete. 2024-05-09T05:01:39.886 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:01:39.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.994+0000 7fae18189640 1 Processor -- start 2024-05-09T05:01:39.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.994+0000 7fae18189640 1 -- start start 2024-05-09T05:01:39.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.995+0000 7fae18189640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fae1015d080 0x7fae1009d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:39.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.995+0000 7fae18189640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fae1009ba20 0x7fae100b2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:39.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.995+0000 7fae18189640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae10070510 con 0x7fae1006f230 2024-05-09T05:01:39.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.995+0000 7fae18189640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fae1009ab70 con 0x7fae1015d080 2024-05-09T05:01:39.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.995+0000 7fae18189640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fae100707d0 con 0x7fae1009ba20 2024-05-09T05:01:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.995+0000 7fae15f25640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fae1015d080 0x7fae1009d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.995+0000 7fae15724640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fae1006f230 0x7fae1009b310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57728/0 (socket says 172.21.15.60:57728) 2024-05-09T05:01:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.995+0000 7fae15724640 1 -- 172.21.15.60:0/875582464 learned_addr learned my addr 172.21.15.60:0/875582464 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.995+0000 7fae16726640 1 --2- 172.21.15.60:0/875582464 >> v2:172.21.15.114:3300/0 conn(0x7fae1009ba20 0x7fae100b2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.995+0000 7fae15f25640 1 -- 172.21.15.60:0/875582464 >> v2:172.21.15.114:3300/0 conn(0x7fae1009ba20 msgr2=0x7fae100b2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.995+0000 7fae15f25640 1 --2- 172.21.15.60:0/875582464 >> v2:172.21.15.114:3300/0 conn(0x7fae1009ba20 0x7fae100b2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.995+0000 7fae15f25640 1 -- 172.21.15.60:0/875582464 >> v1:172.21.15.60:6789/0 conn(0x7fae1006f230 legacy=0x7fae1009b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.995+0000 7fae14f23640 1 -- 172.21.15.60:0/875582464 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1151398892 0 0) 0x7fae10070510 con 0x7fae1006f230 2024-05-09T05:01:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.995+0000 7fae15f25640 1 -- 172.21.15.60:0/875582464 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae10070ab0 con 0x7fae1015d080 2024-05-09T05:01:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.995+0000 7fae16726640 1 --2- 172.21.15.60:0/875582464 >> v2:172.21.15.114:3300/0 conn(0x7fae1009ba20 0x7fae100b2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:40.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.996+0000 7fae15f25640 1 --2- 172.21.15.60:0/875582464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1015d080 0x7fae1009d520 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fae100ab000 tx=0x7fae0c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4836dfadf45402f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:40.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.996+0000 7fae14f23640 1 -- 172.21.15.60:0/875582464 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0c037070 con 0x7fae1015d080 2024-05-09T05:01:40.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.996+0000 7fae14f23640 1 -- 172.21.15.60:0/875582464 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae0c032440 con 0x7fae1015d080 2024-05-09T05:01:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.996+0000 7fae14f23640 1 -- 172.21.15.60:0/875582464 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0c032810 con 0x7fae1015d080 2024-05-09T05:01:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.997+0000 7fae18189640 1 -- 172.21.15.60:0/875582464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1015d080 msgr2=0x7fae1009d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.997+0000 7fae18189640 1 --2- 172.21.15.60:0/875582464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1015d080 0x7fae1009d520 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fae100ab000 tx=0x7fae0c02a1a0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.997+0000 7fae18189640 1 -- 172.21.15.60:0/875582464 shutdown_connections 2024-05-09T05:01:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.997+0000 7fae18189640 1 --2- 172.21.15.60:0/875582464 >> v2:172.21.15.114:3300/0 conn(0x7fae1009ba20 0x7fae100b2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.997+0000 7fae18189640 1 --2- 172.21.15.60:0/875582464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1015d080 0x7fae1009d520 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.997+0000 7fae18189640 1 -- 172.21.15.60:0/875582464 >> 172.21.15.60:0/875582464 conn(0x7fae10072390 msgr2=0x7fae100727a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.997+0000 7fae18189640 1 -- 172.21.15.60:0/875582464 shutdown_connections 2024-05-09T05:01:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.997+0000 7fae18189640 1 -- 172.21.15.60:0/875582464 wait complete. 2024-05-09T05:01:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.998+0000 7fae18189640 1 Processor -- start 2024-05-09T05:01:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.998+0000 7fae18189640 1 -- start start 2024-05-09T05:01:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.999+0000 7fae18189640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae1006f230 0x7fae101f8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:40.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.999+0000 7fae18189640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae1009ba20 0x7fae1015d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:40.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.999+0000 7fae18189640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1015d080 0x7fae1006cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:40.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.999+0000 7fae18189640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fae1009a770 con 0x7fae1015d080 2024-05-09T05:01:40.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.999+0000 7fae18189640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fae1015d460 con 0x7fae1009ba20 2024-05-09T05:01:40.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.999+0000 7fae18189640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fae1009a8f0 con 0x7fae1006f230 2024-05-09T05:01:40.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.999+0000 7fae16726640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1015d080 0x7fae1006cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:40.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.999+0000 7fae16726640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1015d080 0x7fae1006cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37612/0 (socket says 172.21.15.60:37612) 2024-05-09T05:01:40.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.999+0000 7fae16726640 1 -- 172.21.15.60:0/39928563 learned_addr learned my addr 172.21.15.60:0/39928563 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:40.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.999+0000 7fae15f25640 1 --2- 172.21.15.60:0/39928563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae1006f230 0x7fae101f8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:40.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.999+0000 7fae15724640 1 --2- 172.21.15.60:0/39928563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae1009ba20 0x7fae1015d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:40.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.999+0000 7fae16726640 1 -- 172.21.15.60:0/39928563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae1006f230 msgr2=0x7fae101f8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:40.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.999+0000 7fae16726640 1 --2- 172.21.15.60:0/39928563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae1006f230 0x7fae101f8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.999+0000 7fae16726640 1 -- 172.21.15.60:0/39928563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae1009ba20 msgr2=0x7fae1015d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:40.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.999+0000 7fae16726640 1 --2- 172.21.15.60:0/39928563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae1009ba20 0x7fae1015d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.999+0000 7fae16726640 1 -- 172.21.15.60:0/39928563 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae10157f80 con 0x7fae1015d080 2024-05-09T05:01:40.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:39.999+0000 7fae15724640 1 --2- 172.21.15.60:0/39928563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae1009ba20 0x7fae1015d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:40.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.000+0000 7fae16726640 1 --2- 172.21.15.60:0/39928563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1015d080 0x7fae1006cd60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fae08007ef0 tx=0x7fae0800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:40.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.000+0000 7fae067fc640 1 -- 172.21.15.60:0/39928563 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0801a070 con 0x7fae1015d080 2024-05-09T05:01:40.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.000+0000 7fae067fc640 1 -- 172.21.15.60:0/39928563 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae08003b90 con 0x7fae1015d080 2024-05-09T05:01:40.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.000+0000 7fae18189640 1 -- 172.21.15.60:0/39928563 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae10158260 con 0x7fae1015d080 2024-05-09T05:01:40.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.000+0000 7fae067fc640 1 -- 172.21.15.60:0/39928563 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae08004030 con 0x7fae1015d080 2024-05-09T05:01:40.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.000+0000 7fae18189640 1 -- 172.21.15.60:0/39928563 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae100ad140 con 0x7fae1015d080 2024-05-09T05:01:40.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.001+0000 7fae18189640 1 -- 172.21.15.60:0/39928563 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae100af8a0 con 0x7fae1015d080 2024-05-09T05:01:40.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.001+0000 7fae067fc640 1 -- 172.21.15.60:0/39928563 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fae0801d070 con 0x7fae1015d080 2024-05-09T05:01:40.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.002+0000 7fae067fc640 1 --2- 172.21.15.60:0/39928563 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fadf40762d0 0x7fadf4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:40.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.002+0000 7fae067fc640 1 -- 172.21.15.60:0/39928563 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6101+0+0 (secure 0 0 0) 0x7fae08097960 con 0x7fae1015d080 2024-05-09T05:01:40.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.002+0000 7fae15f25640 1 --2- 172.21.15.60:0/39928563 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fadf40762d0 0x7fadf4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:40.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.003+0000 7fae15f25640 1 --2- 172.21.15.60:0/39928563 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fadf40762d0 0x7fadf4078790 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fae0c034000 tx=0x7fae0c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:40.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.006+0000 7fae067fc640 1 -- 172.21.15.60:0/39928563 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fae100af8a0 con 0x7fae1015d080 2024-05-09T05:01:40.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.202+0000 7fae18189640 1 -- 172.21.15.60:0/39928563 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fae10152d90 con 0x7fae1015d080 2024-05-09T05:01:40.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.203+0000 7fae067fc640 1 -- 172.21.15.60:0/39928563 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+16927 (secure 0 0 0) 0x7fae10152d90 con 0x7fae1015d080 2024-05-09T05:01:40.204 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:40.204 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":50,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:01:39.237537+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:01:36.910324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T05:01:38.185514+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:01:40.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.205+0000 7fae18189640 1 -- 172.21.15.60:0/39928563 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fadf40762d0 msgr2=0x7fadf4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:40.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.205+0000 7fae18189640 1 --2- 172.21.15.60:0/39928563 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fadf40762d0 0x7fadf4078790 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fae0c034000 tx=0x7fae0c02e000 comp rx=0 tx=0).stop 2024-05-09T05:01:40.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.205+0000 7fae18189640 1 -- 172.21.15.60:0/39928563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1015d080 msgr2=0x7fae1006cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:40.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.205+0000 7fae18189640 1 --2- 172.21.15.60:0/39928563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1015d080 0x7fae1006cd60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fae08007ef0 tx=0x7fae0800d4a0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.206+0000 7fae18189640 1 -- 172.21.15.60:0/39928563 shutdown_connections 2024-05-09T05:01:40.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.206+0000 7fae18189640 1 --2- 172.21.15.60:0/39928563 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fadf40762d0 0x7fadf4078790 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.206+0000 7fae18189640 1 --2- 172.21.15.60:0/39928563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fae1015d080 0x7fae1006cd60 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.206+0000 7fae18189640 1 --2- 172.21.15.60:0/39928563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae1009ba20 0x7fae1015d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.206+0000 7fae18189640 1 --2- 172.21.15.60:0/39928563 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fae1006f230 0x7fae101f8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.206+0000 7fae18189640 1 -- 172.21.15.60:0/39928563 >> 172.21.15.60:0/39928563 conn(0x7fae10072390 msgr2=0x7fae10151e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:40.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.206+0000 7fae18189640 1 -- 172.21.15.60:0/39928563 shutdown_connections 2024-05-09T05:01:40.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.206+0000 7fae18189640 1 -- 172.21.15.60:0/39928563 wait complete. 2024-05-09T05:01:40.218 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:01:40.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.321+0000 7fd76b412640 1 Processor -- start 2024-05-09T05:01:40.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.321+0000 7fd76b412640 1 -- start start 2024-05-09T05:01:40.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.322+0000 7fd76b412640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd76406fe20 0x7fd7640702d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:40.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.322+0000 7fd76b412640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7640d08e0 con 0x7fd7640736d0 2024-05-09T05:01:40.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.322+0000 7fd76b412640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7640d0bc0 con 0x7fd76415f300 2024-05-09T05:01:40.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.322+0000 7fd76b412640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fd764073240 con 0x7fd76406fe20 2024-05-09T05:01:40.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.322+0000 7fd7689ad640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd76406fe20 0x7fd7640702d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:40.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.322+0000 7fd7699af640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd7640736d0 0x7fd764073b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57734/0 (socket says 172.21.15.60:57734) 2024-05-09T05:01:40.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.322+0000 7fd7689ad640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd76406fe20 0x7fd7640702d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37628/0 (socket says 172.21.15.60:37628) 2024-05-09T05:01:40.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.322+0000 7fd7699af640 1 -- 172.21.15.60:0/2083451317 learned_addr learned my addr 172.21.15.60:0/2083451317 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:40.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.322+0000 7fd7689ad640 1 -- 172.21.15.60:0/2083451317 >> v1:172.21.15.114:6789/0 conn(0x7fd76415f300 legacy=0x7fd764072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:40.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.322+0000 7fd7689ad640 1 -- 172.21.15.60:0/2083451317 >> v1:172.21.15.60:6789/0 conn(0x7fd7640736d0 legacy=0x7fd764073b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:40.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.322+0000 7fd7689ad640 1 -- 172.21.15.60:0/2083451317 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd764072ed0 con 0x7fd76406fe20 2024-05-09T05:01:40.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.323+0000 7fd7689ad640 1 --2- 172.21.15.60:0/2083451317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd76406fe20 0x7fd7640702d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd7540097a0 tx=0x7fd75402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b9851364bce923 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:40.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.323+0000 7fd75bfff640 1 -- 172.21.15.60:0/2083451317 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd754037070 con 0x7fd76406fe20 2024-05-09T05:01:40.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.323+0000 7fd75bfff640 1 -- 172.21.15.60:0/2083451317 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd754032440 con 0x7fd76406fe20 2024-05-09T05:01:40.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.323+0000 7fd75bfff640 1 -- 172.21.15.60:0/2083451317 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd754032810 con 0x7fd76406fe20 2024-05-09T05:01:40.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.324+0000 7fd76b412640 1 -- 172.21.15.60:0/2083451317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd76406fe20 msgr2=0x7fd7640702d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:40.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.324+0000 7fd76b412640 1 --2- 172.21.15.60:0/2083451317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd76406fe20 0x7fd7640702d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd7540097a0 tx=0x7fd75402a230 comp rx=0 tx=0).stop 2024-05-09T05:01:40.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.324+0000 7fd76b412640 1 -- 172.21.15.60:0/2083451317 shutdown_connections 2024-05-09T05:01:40.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.324+0000 7fd76b412640 1 --2- 172.21.15.60:0/2083451317 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd76406fe20 0x7fd7640702d0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.324+0000 7fd76b412640 1 -- 172.21.15.60:0/2083451317 >> 172.21.15.60:0/2083451317 conn(0x7fd7640b4ef0 msgr2=0x7fd76406e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:40.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.324+0000 7fd76b412640 1 -- 172.21.15.60:0/2083451317 shutdown_connections 2024-05-09T05:01:40.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.324+0000 7fd76b412640 1 -- 172.21.15.60:0/2083451317 wait complete. 2024-05-09T05:01:40.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.325+0000 7fd76b412640 1 Processor -- start 2024-05-09T05:01:40.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.325+0000 7fd76b412640 1 -- start start 2024-05-09T05:01:40.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.326+0000 7fd76b412640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd76406fe20 0x7fd76415a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:40.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.326+0000 7fd76b412640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7640736d0 0x7fd7640afd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:40.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.326+0000 7fd76b412640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd76415f300 0x7fd7640d3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:40.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.326+0000 7fd76b412640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd7640d0670 con 0x7fd7640736d0 2024-05-09T05:01:40.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.326+0000 7fd76b412640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd7640d04f0 con 0x7fd76415f300 2024-05-09T05:01:40.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.326+0000 7fd76b412640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd76415f6e0 con 0x7fd76406fe20 2024-05-09T05:01:40.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.326+0000 7fd7689ad640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7640736d0 0x7fd7640afd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:40.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.326+0000 7fd7689ad640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7640736d0 0x7fd7640afd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37640/0 (socket says 172.21.15.60:37640) 2024-05-09T05:01:40.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.326+0000 7fd7689ad640 1 -- 172.21.15.60:0/2390499909 learned_addr learned my addr 172.21.15.60:0/2390499909 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:40.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.326+0000 7fd7699af640 1 --2- 172.21.15.60:0/2390499909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd76415f300 0x7fd7640d3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:40.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.326+0000 7fd7691ae640 1 --2- 172.21.15.60:0/2390499909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd76406fe20 0x7fd76415a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:40.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.326+0000 7fd7689ad640 1 -- 172.21.15.60:0/2390499909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd76406fe20 msgr2=0x7fd76415a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:40.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.326+0000 7fd7689ad640 1 --2- 172.21.15.60:0/2390499909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd76406fe20 0x7fd76415a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.326+0000 7fd7689ad640 1 -- 172.21.15.60:0/2390499909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd76415f300 msgr2=0x7fd7640d3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:40.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.326+0000 7fd7689ad640 1 --2- 172.21.15.60:0/2390499909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd76415f300 0x7fd7640d3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.326+0000 7fd7689ad640 1 -- 172.21.15.60:0/2390499909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7640b0760 con 0x7fd7640736d0 2024-05-09T05:01:40.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.326+0000 7fd7699af640 1 --2- 172.21.15.60:0/2390499909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd76415f300 0x7fd7640d3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:40.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.327+0000 7fd7689ad640 1 --2- 172.21.15.60:0/2390499909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7640736d0 0x7fd7640afd10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd754035040 tx=0x7fd7540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:40.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.327+0000 7fd759ffb640 1 -- 172.21.15.60:0/2390499909 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd754037040 con 0x7fd7640736d0 2024-05-09T05:01:40.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.327+0000 7fd759ffb640 1 -- 172.21.15.60:0/2390499909 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd754030770 con 0x7fd7640736d0 2024-05-09T05:01:40.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.327+0000 7fd76b412640 1 -- 172.21.15.60:0/2390499909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7640ad410 con 0x7fd7640736d0 2024-05-09T05:01:40.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.327+0000 7fd759ffb640 1 -- 172.21.15.60:0/2390499909 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd754030b80 con 0x7fd7640736d0 2024-05-09T05:01:40.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.327+0000 7fd76b412640 1 -- 172.21.15.60:0/2390499909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7640b0a40 con 0x7fd7640736d0 2024-05-09T05:01:40.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.328+0000 7fd76b412640 1 -- 172.21.15.60:0/2390499909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd764051040 con 0x7fd7640736d0 2024-05-09T05:01:40.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.329+0000 7fd759ffb640 1 -- 172.21.15.60:0/2390499909 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd754044020 con 0x7fd7640736d0 2024-05-09T05:01:40.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.329+0000 7fd759ffb640 1 --2- 172.21.15.60:0/2390499909 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd7480762d0 0x7fd748078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:40.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.329+0000 7fd759ffb640 1 -- 172.21.15.60:0/2390499909 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6101+0+0 (secure 0 0 0) 0x7fd7540b7cf0 con 0x7fd7640736d0 2024-05-09T05:01:40.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.330+0000 7fd7691ae640 1 --2- 172.21.15.60:0/2390499909 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd7480762d0 0x7fd748078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:40.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.330+0000 7fd7691ae640 1 --2- 172.21.15.60:0/2390499909 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd7480762d0 0x7fd748078790 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd764161140 tx=0x7fd760009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:40.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.333+0000 7fd759ffb640 1 -- 172.21.15.60:0/2390499909 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd7540bc070 con 0x7fd7640736d0 2024-05-09T05:01:40.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.526+0000 7fd76b412640 1 -- 172.21.15.60:0/2390499909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd7640687a0 con 0x7fd7640736d0 2024-05-09T05:01:40.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.527+0000 7fd759ffb640 1 -- 172.21.15.60:0/2390499909 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) ==== 74+0+16927 (secure 0 0 0) 0x7fd754032e20 con 0x7fd7640736d0 2024-05-09T05:01:40.528 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:40.528 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":50,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:01:39.237537+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:01:36.910324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T05:01:38.185514+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:01:40.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.529+0000 7fd76b412640 1 -- 172.21.15.60:0/2390499909 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd7480762d0 msgr2=0x7fd748078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:40.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.529+0000 7fd76b412640 1 --2- 172.21.15.60:0/2390499909 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd7480762d0 0x7fd748078790 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd764161140 tx=0x7fd760009040 comp rx=0 tx=0).stop 2024-05-09T05:01:40.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.529+0000 7fd76b412640 1 -- 172.21.15.60:0/2390499909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7640736d0 msgr2=0x7fd7640afd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:40.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.529+0000 7fd76b412640 1 --2- 172.21.15.60:0/2390499909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7640736d0 0x7fd7640afd10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd754035040 tx=0x7fd7540046c0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.529+0000 7fd76b412640 1 -- 172.21.15.60:0/2390499909 shutdown_connections 2024-05-09T05:01:40.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.529+0000 7fd76b412640 1 --2- 172.21.15.60:0/2390499909 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd7480762d0 0x7fd748078790 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.529+0000 7fd76b412640 1 --2- 172.21.15.60:0/2390499909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd76415f300 0x7fd7640d3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.529+0000 7fd76b412640 1 --2- 172.21.15.60:0/2390499909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7640736d0 0x7fd7640afd10 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.529+0000 7fd76b412640 1 --2- 172.21.15.60:0/2390499909 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd76406fe20 0x7fd76415a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.529+0000 7fd76b412640 1 -- 172.21.15.60:0/2390499909 >> 172.21.15.60:0/2390499909 conn(0x7fd7640b4ef0 msgr2=0x7fd76409d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:40.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.529+0000 7fd76b412640 1 -- 172.21.15.60:0/2390499909 shutdown_connections 2024-05-09T05:01:40.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.529+0000 7fd76b412640 1 -- 172.21.15.60:0/2390499909 wait complete. 2024-05-09T05:01:40.541 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-09T05:01:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.647+0000 7fd93695f640 1 Processor -- start 2024-05-09T05:01:40.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.647+0000 7fd93695f640 1 -- start start 2024-05-09T05:01:40.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.648+0000 7fd93695f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd93015f3c0 0x7fd930162e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:40.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.648+0000 7fd93695f640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd9300b9d00 0x7fd9300732d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:40.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.648+0000 7fd93695f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd930163740 con 0x7fd9300b7fe0 2024-05-09T05:01:40.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.648+0000 7fd93695f640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fd930163a20 con 0x7fd9300b9d00 2024-05-09T05:01:40.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.648+0000 7fd93695f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd930163d00 con 0x7fd93015f3c0 2024-05-09T05:01:40.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.648+0000 7fd934efc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd9300b9d00 0x7fd9300732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.648+0000 7fd934efc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd9300b9d00 0x7fd9300732d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37648/0 (socket says 172.21.15.60:37648) 2024-05-09T05:01:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.648+0000 7fd934efc640 1 -- 172.21.15.60:0/3219212697 learned_addr learned my addr 172.21.15.60:0/3219212697 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.648+0000 7fd92ffff640 1 --2- 172.21.15.60:0/3219212697 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd93015f3c0 0x7fd930162e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.649+0000 7fd934efc640 1 -- 172.21.15.60:0/3219212697 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd93015f3c0 msgr2=0x7fd930162e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.649+0000 7fd934efc640 1 --2- 172.21.15.60:0/3219212697 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd93015f3c0 0x7fd930162e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.649+0000 7fd934efc640 1 -- 172.21.15.60:0/3219212697 >> v1:172.21.15.114:6789/0 conn(0x7fd9300b7fe0 legacy=0x7fd9300b95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.649+0000 7fd934efc640 1 -- 172.21.15.60:0/3219212697 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd930072ad0 con 0x7fd9300b9d00 2024-05-09T05:01:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.649+0000 7fd934efc640 1 --2- 172.21.15.60:0/3219212697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9300b9d00 0x7fd9300732d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd928009e80 tx=0x7fd92802a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=901b76a935ab2736 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.650+0000 7fd92f7fe640 1 -- 172.21.15.60:0/3219212697 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd928036070 con 0x7fd9300b9d00 2024-05-09T05:01:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.650+0000 7fd92f7fe640 1 -- 172.21.15.60:0/3219212697 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd928031600 con 0x7fd9300b9d00 2024-05-09T05:01:40.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.650+0000 7fd92f7fe640 1 -- 172.21.15.60:0/3219212697 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9280319d0 con 0x7fd9300b9d00 2024-05-09T05:01:40.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.650+0000 7fd93695f640 1 -- 172.21.15.60:0/3219212697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9300b9d00 msgr2=0x7fd9300732d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:40.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.650+0000 7fd93695f640 1 --2- 172.21.15.60:0/3219212697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9300b9d00 0x7fd9300732d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd928009e80 tx=0x7fd92802a750 comp rx=0 tx=0).stop 2024-05-09T05:01:40.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.650+0000 7fd93695f640 1 -- 172.21.15.60:0/3219212697 shutdown_connections 2024-05-09T05:01:40.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.650+0000 7fd93695f640 1 --2- 172.21.15.60:0/3219212697 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9300b9d00 0x7fd9300732d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.650+0000 7fd93695f640 1 --2- 172.21.15.60:0/3219212697 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd93015f3c0 0x7fd930162e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.650+0000 7fd93695f640 1 -- 172.21.15.60:0/3219212697 >> 172.21.15.60:0/3219212697 conn(0x7fd93006cae0 msgr2=0x7fd9300d53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:40.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.650+0000 7fd93695f640 1 -- 172.21.15.60:0/3219212697 shutdown_connections 2024-05-09T05:01:40.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.650+0000 7fd93695f640 1 -- 172.21.15.60:0/3219212697 wait complete. 2024-05-09T05:01:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.652+0000 7fd93695f640 1 Processor -- start 2024-05-09T05:01:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.652+0000 7fd93695f640 1 -- start start 2024-05-09T05:01:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.652+0000 7fd93695f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9300b7fe0 0x7fd9300ae890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.652+0000 7fd93695f640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9300b9d00 0x7fd9301eb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.652+0000 7fd93695f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd93015f3c0 0x7fd93015a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.652+0000 7fd93695f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fd930163380 con 0x7fd9300b7fe0 2024-05-09T05:01:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.652+0000 7fd93695f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd9300d1dc0 con 0x7fd93015f3c0 2024-05-09T05:01:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.652+0000 7fd93695f640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fd930163500 con 0x7fd9300b9d00 2024-05-09T05:01:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.652+0000 7fd92ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9300b7fe0 0x7fd9300ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.652+0000 7fd92ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9300b7fe0 0x7fd9300ae890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37656/0 (socket says 172.21.15.60:37656) 2024-05-09T05:01:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.652+0000 7fd92ffff640 1 -- 172.21.15.60:0/3907063743 learned_addr learned my addr 172.21.15.60:0/3907063743 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.653+0000 7fd927fff640 1 --2- 172.21.15.60:0/3907063743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9300b9d00 0x7fd9301eb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.653+0000 7fd934efc640 1 --2- 172.21.15.60:0/3907063743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd93015f3c0 0x7fd93015a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:40.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.653+0000 7fd92ffff640 1 -- 172.21.15.60:0/3907063743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9300b9d00 msgr2=0x7fd9301eb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:40.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.653+0000 7fd92ffff640 1 --2- 172.21.15.60:0/3907063743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9300b9d00 0x7fd9301eb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.653+0000 7fd92ffff640 1 -- 172.21.15.60:0/3907063743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd93015f3c0 msgr2=0x7fd93015a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:40.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.653+0000 7fd92ffff640 1 --2- 172.21.15.60:0/3907063743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd93015f3c0 0x7fd93015a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:40.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.653+0000 7fd92ffff640 1 -- 172.21.15.60:0/3907063743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd930162330 con 0x7fd9300b7fe0 2024-05-09T05:01:40.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.653+0000 7fd927fff640 1 --2- 172.21.15.60:0/3907063743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9300b9d00 0x7fd9301eb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:40.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.653+0000 7fd934efc640 1 --2- 172.21.15.60:0/3907063743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd93015f3c0 0x7fd93015a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:40.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.653+0000 7fd92ffff640 1 --2- 172.21.15.60:0/3907063743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9300b7fe0 0x7fd9300ae890 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd92000e870 tx=0x7fd92000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:40.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.653+0000 7fd92d7fa640 1 -- 172.21.15.60:0/3907063743 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd920017070 con 0x7fd9300b7fe0 2024-05-09T05:01:40.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.653+0000 7fd92d7fa640 1 -- 172.21.15.60:0/3907063743 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd92000c940 con 0x7fd9300b7fe0 2024-05-09T05:01:40.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.653+0000 7fd92d7fa640 1 -- 172.21.15.60:0/3907063743 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92000cd30 con 0x7fd9300b7fe0 2024-05-09T05:01:40.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.653+0000 7fd93695f640 1 -- 172.21.15.60:0/3907063743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd930162610 con 0x7fd9300b7fe0 2024-05-09T05:01:40.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.653+0000 7fd93695f640 1 -- 172.21.15.60:0/3907063743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9301eab10 con 0x7fd9300b7fe0 2024-05-09T05:01:40.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.654+0000 7fd93695f640 1 -- 172.21.15.60:0/3907063743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd93015b0c0 con 0x7fd9300b7fe0 2024-05-09T05:01:40.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.655+0000 7fd92d7fa640 1 -- 172.21.15.60:0/3907063743 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd92001a070 con 0x7fd9300b7fe0 2024-05-09T05:01:40.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.655+0000 7fd92d7fa640 1 --2- 172.21.15.60:0/3907063743 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd9100762d0 0x7fd910078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:40.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.655+0000 7fd92d7fa640 1 -- 172.21.15.60:0/3907063743 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6101+0+0 (secure 0 0 0) 0x7fd920097950 con 0x7fd9300b7fe0 2024-05-09T05:01:40.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.655+0000 7fd927fff640 1 --2- 172.21.15.60:0/3907063743 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd9100762d0 0x7fd910078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:40.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.656+0000 7fd927fff640 1 --2- 172.21.15.60:0/3907063743 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd9100762d0 0x7fd910078790 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd93015b780 tx=0x7fd918009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:40.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.660+0000 7fd92d7fa640 1 -- 172.21.15.60:0/3907063743 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd9200611b0 con 0x7fd9300b7fe0 2024-05-09T05:01:40.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:40.892+0000 7fd93695f640 1 -- 172.21.15.60:0/3907063743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fd930051040 con 0x7fd9300b7fe0 2024-05-09T05:01:41.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:41.879+0000 7fd92d7fa640 1 -- 172.21.15.60:0/3907063743 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v26) ==== 118+0+0 (secure 0 0 0) 0x7fd920060ab0 con 0x7fd9300b7fe0 2024-05-09T05:01:41.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:41.882+0000 7fd93695f640 1 -- 172.21.15.60:0/3907063743 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd9100762d0 msgr2=0x7fd910078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:41.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:41.882+0000 7fd93695f640 1 --2- 172.21.15.60:0/3907063743 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd9100762d0 0x7fd910078790 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd93015b780 tx=0x7fd918009040 comp rx=0 tx=0).stop 2024-05-09T05:01:41.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:41.882+0000 7fd93695f640 1 -- 172.21.15.60:0/3907063743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9300b7fe0 msgr2=0x7fd9300ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:41.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:41.882+0000 7fd93695f640 1 --2- 172.21.15.60:0/3907063743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9300b7fe0 0x7fd9300ae890 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd92000e870 tx=0x7fd92000ed30 comp rx=0 tx=0).stop 2024-05-09T05:01:41.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:41.882+0000 7fd93695f640 1 -- 172.21.15.60:0/3907063743 shutdown_connections 2024-05-09T05:01:41.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:41.882+0000 7fd93695f640 1 --2- 172.21.15.60:0/3907063743 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fd9100762d0 0x7fd910078790 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:41.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:41.882+0000 7fd93695f640 1 --2- 172.21.15.60:0/3907063743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd93015f3c0 0x7fd93015a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:41.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:41.882+0000 7fd93695f640 1 --2- 172.21.15.60:0/3907063743 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fd9300b9d00 0x7fd9301eb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:41.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:41.882+0000 7fd93695f640 1 --2- 172.21.15.60:0/3907063743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9300b7fe0 0x7fd9300ae890 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:41.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:41.882+0000 7fd93695f640 1 -- 172.21.15.60:0/3907063743 >> 172.21.15.60:0/3907063743 conn(0x7fd93006cae0 msgr2=0x7fd930072ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:41.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:41.882+0000 7fd93695f640 1 -- 172.21.15.60:0/3907063743 shutdown_connections 2024-05-09T05:01:41.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:41.883+0000 7fd93695f640 1 -- 172.21.15.60:0/3907063743 wait complete. 2024-05-09T05:01:41.897 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-09T05:01:41.897 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-09T05:01:42.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.010+0000 7f9e0d67e640 1 Processor -- start 2024-05-09T05:01:42.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.010+0000 7f9e0d67e640 1 -- start start 2024-05-09T05:01:42.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.011+0000 7f9e0d67e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e080b65d0 0x7f9e080b7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:42.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.011+0000 7f9e0d67e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9e08162fa0 0x7f9e080730e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:42.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.011+0000 7f9e0d67e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e08072c00 con 0x7f9e0815f330 2024-05-09T05:01:42.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.011+0000 7f9e0d67e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f9e08163540 con 0x7f9e08162fa0 2024-05-09T05:01:42.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.011+0000 7f9e0d67e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9e08163820 con 0x7f9e080b65d0 2024-05-09T05:01:42.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.011+0000 7f9e077fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9e08162fa0 0x7f9e080730e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:42.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.011+0000 7f9e077fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9e08162fa0 0x7f9e080730e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37666/0 (socket says 172.21.15.60:37666) 2024-05-09T05:01:42.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.011+0000 7f9e077fe640 1 -- 172.21.15.60:0/1493071579 learned_addr learned my addr 172.21.15.60:0/1493071579 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:42.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.011+0000 7f9dfffff640 1 --2- 172.21.15.60:0/1493071579 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e080b65d0 0x7f9e080b7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:42.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.011+0000 7f9e077fe640 1 -- 172.21.15.60:0/1493071579 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e080b65d0 msgr2=0x7f9e080b7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:42.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.011+0000 7f9e077fe640 1 --2- 172.21.15.60:0/1493071579 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e080b65d0 0x7f9e080b7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:42.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.011+0000 7f9e077fe640 1 -- 172.21.15.60:0/1493071579 >> v1:172.21.15.114:6789/0 conn(0x7f9e0815f330 legacy=0x7f9e080b5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:42.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.011+0000 7f9e077fe640 1 -- 172.21.15.60:0/1493071579 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e08163b00 con 0x7f9e08162fa0 2024-05-09T05:01:42.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.012+0000 7f9e077fe640 1 --2- 172.21.15.60:0/1493071579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e08162fa0 0x7f9e080730e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009e50 tx=0x7f9e0002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f71af90e3f33a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:42.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.012+0000 7f9e067fc640 1 -- 172.21.15.60:0/1493071579 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e00036070 con 0x7f9e08162fa0 2024-05-09T05:01:42.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.012+0000 7f9e067fc640 1 -- 172.21.15.60:0/1493071579 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e00031600 con 0x7f9e08162fa0 2024-05-09T05:01:42.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.012+0000 7f9e067fc640 1 -- 172.21.15.60:0/1493071579 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e000319d0 con 0x7f9e08162fa0 2024-05-09T05:01:42.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.013+0000 7f9e0d67e640 1 -- 172.21.15.60:0/1493071579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e08162fa0 msgr2=0x7f9e080730e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:42.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.013+0000 7f9e0d67e640 1 --2- 172.21.15.60:0/1493071579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e08162fa0 0x7f9e080730e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009e50 tx=0x7f9e0002a710 comp rx=0 tx=0).stop 2024-05-09T05:01:42.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.013+0000 7f9e0d67e640 1 -- 172.21.15.60:0/1493071579 shutdown_connections 2024-05-09T05:01:42.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.013+0000 7f9e0d67e640 1 --2- 172.21.15.60:0/1493071579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e08162fa0 0x7f9e080730e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:42.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.013+0000 7f9e0d67e640 1 --2- 172.21.15.60:0/1493071579 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e080b65d0 0x7f9e080b7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:42.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.013+0000 7f9e0d67e640 1 -- 172.21.15.60:0/1493071579 >> 172.21.15.60:0/1493071579 conn(0x7f9e0806ca80 msgr2=0x7f9e080d53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:42.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.013+0000 7f9e0d67e640 1 -- 172.21.15.60:0/1493071579 shutdown_connections 2024-05-09T05:01:42.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.013+0000 7f9e0d67e640 1 -- 172.21.15.60:0/1493071579 wait complete. 2024-05-09T05:01:42.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.014+0000 7f9e0d67e640 1 Processor -- start 2024-05-09T05:01:42.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.014+0000 7f9e0d67e640 1 -- start start 2024-05-09T05:01:42.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.015+0000 7f9e0d67e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e080b65d0 0x7f9e0815deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:42.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.015+0000 7f9e0d67e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e0815f330 0x7f9e0815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:42.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.015+0000 7f9e0d67e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e08162fa0 0x7f9e0815c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:42.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.015+0000 7f9e0d67e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0815f710 con 0x7f9e080b65d0 2024-05-09T05:01:42.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.015+0000 7f9e0d67e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9e08163de0 con 0x7f9e0815f330 2024-05-09T05:01:42.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.015+0000 7f9e0d67e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9e0815eb70 con 0x7f9e08162fa0 2024-05-09T05:01:42.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.015+0000 7f9e06ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e080b65d0 0x7f9e0815deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:42.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.015+0000 7f9e06ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e080b65d0 0x7f9e0815deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37676/0 (socket says 172.21.15.60:37676) 2024-05-09T05:01:42.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.015+0000 7f9e06ffd640 1 -- 172.21.15.60:0/2146047761 learned_addr learned my addr 172.21.15.60:0/2146047761 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:42.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.015+0000 7f9e077fe640 1 --2- 172.21.15.60:0/2146047761 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e08162fa0 0x7f9e0815c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:42.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.015+0000 7f9dfffff640 1 --2- 172.21.15.60:0/2146047761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e0815f330 0x7f9e0815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:42.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.015+0000 7f9e06ffd640 1 -- 172.21.15.60:0/2146047761 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e08162fa0 msgr2=0x7f9e0815c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:42.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.015+0000 7f9e06ffd640 1 --2- 172.21.15.60:0/2146047761 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e08162fa0 0x7f9e0815c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:42.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.015+0000 7f9e06ffd640 1 -- 172.21.15.60:0/2146047761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e0815f330 msgr2=0x7f9e0815ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:42.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.015+0000 7f9e06ffd640 1 --2- 172.21.15.60:0/2146047761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e0815f330 0x7f9e0815ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:42.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.015+0000 7f9e06ffd640 1 -- 172.21.15.60:0/2146047761 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e0815e3f0 con 0x7f9e080b65d0 2024-05-09T05:01:42.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.015+0000 7f9dfffff640 1 --2- 172.21.15.60:0/2146047761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e0815f330 0x7f9e0815ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:42.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.016+0000 7f9e06ffd640 1 --2- 172.21.15.60:0/2146047761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e080b65d0 0x7f9e0815deb0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9df800c480 tx=0x7f9df800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:42.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.016+0000 7f9dff7fe640 1 -- 172.21.15.60:0/2146047761 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df80052d0 con 0x7f9e080b65d0 2024-05-09T05:01:42.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.016+0000 7f9dff7fe640 1 -- 172.21.15.60:0/2146047761 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9df8005470 con 0x7f9e080b65d0 2024-05-09T05:01:42.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.016+0000 7f9dff7fe640 1 -- 172.21.15.60:0/2146047761 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df8012870 con 0x7f9e080b65d0 2024-05-09T05:01:42.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.016+0000 7f9e0d67e640 1 -- 172.21.15.60:0/2146047761 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e080d2340 con 0x7f9e080b65d0 2024-05-09T05:01:42.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.016+0000 7f9e0d67e640 1 -- 172.21.15.60:0/2146047761 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e080d2900 con 0x7f9e080b65d0 2024-05-09T05:01:42.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.017+0000 7f9e0d67e640 1 -- 172.21.15.60:0/2146047761 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e08051040 con 0x7f9e080b65d0 2024-05-09T05:01:42.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.017+0000 7f9dff7fe640 1 -- 172.21.15.60:0/2146047761 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9df8012a10 con 0x7f9e080b65d0 2024-05-09T05:01:42.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.018+0000 7f9dff7fe640 1 --2- 172.21.15.60:0/2146047761 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9de80762d0 0x7f9de8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:42.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.018+0000 7f9dff7fe640 1 -- 172.21.15.60:0/2146047761 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f9df8097320 con 0x7f9e080b65d0 2024-05-09T05:01:42.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.018+0000 7f9dfffff640 1 --2- 172.21.15.60:0/2146047761 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9de80762d0 0x7f9de8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:42.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.019+0000 7f9dfffff640 1 --2- 172.21.15.60:0/2146047761 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9de80762d0 0x7f9de8078790 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9e0814e780 tx=0x7f9df0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:42.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.022+0000 7f9dff7fe640 1 -- 172.21.15.60:0/2146047761 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9df8060b80 con 0x7f9e080b65d0 2024-05-09T05:01:42.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.251+0000 7f9e0d67e640 1 -- 172.21.15.60:0/2146047761 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f9e0815e620 con 0x7f9e080b65d0 2024-05-09T05:01:42.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.881+0000 7f9dff7fe640 1 -- 172.21.15.60:0/2146047761 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v27) ==== 122+0+0 (secure 0 0 0) 0x7f9df8060480 con 0x7f9e080b65d0 2024-05-09T05:01:42.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.883+0000 7f9e0d67e640 1 -- 172.21.15.60:0/2146047761 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9de80762d0 msgr2=0x7f9de8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:42.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.883+0000 7f9e0d67e640 1 --2- 172.21.15.60:0/2146047761 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9de80762d0 0x7f9de8078790 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9e0814e780 tx=0x7f9df0008040 comp rx=0 tx=0).stop 2024-05-09T05:01:42.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.884+0000 7f9e0d67e640 1 -- 172.21.15.60:0/2146047761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e080b65d0 msgr2=0x7f9e0815deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:42.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.884+0000 7f9e0d67e640 1 --2- 172.21.15.60:0/2146047761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e080b65d0 0x7f9e0815deb0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9df800c480 tx=0x7f9df800c940 comp rx=0 tx=0).stop 2024-05-09T05:01:42.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.884+0000 7f9e0d67e640 1 -- 172.21.15.60:0/2146047761 shutdown_connections 2024-05-09T05:01:42.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.884+0000 7f9e0d67e640 1 --2- 172.21.15.60:0/2146047761 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9de80762d0 0x7f9de8078790 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:42.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.884+0000 7f9e0d67e640 1 --2- 172.21.15.60:0/2146047761 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9e08162fa0 0x7f9e0815c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:42.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.884+0000 7f9e0d67e640 1 --2- 172.21.15.60:0/2146047761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e0815f330 0x7f9e0815ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:42.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.884+0000 7f9e0d67e640 1 --2- 172.21.15.60:0/2146047761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e080b65d0 0x7f9e0815deb0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:42.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.884+0000 7f9e0d67e640 1 -- 172.21.15.60:0/2146047761 >> 172.21.15.60:0/2146047761 conn(0x7f9e0806ca80 msgr2=0x7f9e0806ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:42.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.884+0000 7f9e0d67e640 1 -- 172.21.15.60:0/2146047761 shutdown_connections 2024-05-09T05:01:42.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:42.884+0000 7f9e0d67e640 1 -- 172.21.15.60:0/2146047761 wait complete. 2024-05-09T05:01:42.898 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-09T05:01:42.898 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:43.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.008+0000 7fb6ab42a640 1 Processor -- start 2024-05-09T05:01:43.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.008+0000 7fb6ab42a640 1 -- start start 2024-05-09T05:01:43.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.009+0000 7fb6ab42a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb6a4098d60 0x7fb6a40ae910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.009+0000 7fb6ab42a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6a415fad0 con 0x7fb6a409b640 2024-05-09T05:01:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.009+0000 7fb6ab42a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6a409a880 con 0x7fb6a40b2ea0 2024-05-09T05:01:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.009+0000 7fb6ab42a640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fb6a409ab60 con 0x7fb6a4098d60 2024-05-09T05:01:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.009+0000 7fb6a99c7640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb6a409b640 0x7fb6a409bad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57742/0 (socket says 172.21.15.60:57742) 2024-05-09T05:01:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.009+0000 7fb6a99c7640 1 -- 172.21.15.60:0/871607709 learned_addr learned my addr 172.21.15.60:0/871607709 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.009+0000 7fb6a89c5640 1 --2- 172.21.15.60:0/871607709 >> v2:172.21.15.114:3300/0 conn(0x7fb6a4098d60 0x7fb6a40ae910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.009+0000 7fb69bfff640 1 -- 172.21.15.60:0/871607709 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3951146005 0 0) 0x7fb6a415fad0 con 0x7fb6a409b640 2024-05-09T05:01:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb69bfff640 1 -- 172.21.15.60:0/871607709 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb690003520 con 0x7fb6a409b640 2024-05-09T05:01:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb69bfff640 1 -- 172.21.15.60:0/871607709 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 901307000 0 0) 0x7fb690003520 con 0x7fb6a409b640 2024-05-09T05:01:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb69bfff640 1 -- 172.21.15.60:0/871607709 >> v2:172.21.15.114:3300/0 conn(0x7fb6a4098d60 msgr2=0x7fb6a40ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb69bfff640 1 --2- 172.21.15.60:0/871607709 >> v2:172.21.15.114:3300/0 conn(0x7fb6a4098d60 0x7fb6a40ae910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb69bfff640 1 -- 172.21.15.60:0/871607709 >> v1:172.21.15.114:6789/0 conn(0x7fb6a40b2ea0 legacy=0x7fb6a40b3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb69bfff640 1 -- 172.21.15.60:0/871607709 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a409ae40 con 0x7fb6a409b640 2024-05-09T05:01:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb69bfff640 1 -- 172.21.15.60:0/871607709 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7fb69c001a70 con 0x7fb6a409b640 2024-05-09T05:01:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb69bfff640 1 -- 172.21.15.60:0/871607709 >> v1:172.21.15.60:6789/0 conn(0x7fb6a409b640 legacy=0x7fb6a409bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb69bfff640 1 --2- 172.21.15.60:0/871607709 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb690003a00 0x7fb690003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb69bfff640 1 --2- 172.21.15.60:0/871607709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb690004800 0x7fb690004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb69bfff640 1 --2- 172.21.15.60:0/871607709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6900051c0 0x7fb69000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb69bfff640 1 -- 172.21.15.60:0/871607709 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a409ab60 con 0x7fb690004800 2024-05-09T05:01:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb69bfff640 1 -- 172.21.15.60:0/871607709 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb69000c110 con 0x7fb6900051c0 2024-05-09T05:01:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb69bfff640 1 -- 172.21.15.60:0/871607709 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb69000c310 con 0x7fb690003a00 2024-05-09T05:01:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb6a89c5640 1 --2- 172.21.15.60:0/871607709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb690004800 0x7fb690004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb6a91c6640 1 --2- 172.21.15.60:0/871607709 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb690003a00 0x7fb690003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb6a99c7640 1 --2- 172.21.15.60:0/871607709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6900051c0 0x7fb69000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb6a89c5640 1 -- 172.21.15.60:0/871607709 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb690003a00 msgr2=0x7fb690003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb6a89c5640 1 --2- 172.21.15.60:0/871607709 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb690003a00 0x7fb690003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb6a89c5640 1 -- 172.21.15.60:0/871607709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6900051c0 msgr2=0x7fb69000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:43.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb6a89c5640 1 --2- 172.21.15.60:0/871607709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6900051c0 0x7fb69000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb6a99c7640 1 -- 172.21.15.60:0/871607709 reap_dead start 2024-05-09T05:01:43.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.010+0000 7fb6a89c5640 1 -- 172.21.15.60:0/871607709 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb690003520 con 0x7fb690004800 2024-05-09T05:01:43.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.011+0000 7fb6a89c5640 1 --2- 172.21.15.60:0/871607709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb690004800 0x7fb690004c80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb694009aa0 tx=0x7fb690005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=837e7ccbbc6e87bf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:43.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.011+0000 7fb69bfff640 1 -- 172.21.15.60:0/871607709 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb694036070 con 0x7fb690004800 2024-05-09T05:01:43.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.011+0000 7fb69bfff640 1 -- 172.21.15.60:0/871607709 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb694031400 con 0x7fb690004800 2024-05-09T05:01:43.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.011+0000 7fb69bfff640 1 -- 172.21.15.60:0/871607709 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb694009670 con 0x7fb690004800 2024-05-09T05:01:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.012+0000 7fb6ab42a640 1 -- 172.21.15.60:0/871607709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb690004800 msgr2=0x7fb690004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.012+0000 7fb6ab42a640 1 --2- 172.21.15.60:0/871607709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb690004800 0x7fb690004c80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb694009aa0 tx=0x7fb690005830 comp rx=0 tx=0).stop 2024-05-09T05:01:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.012+0000 7fb6ab42a640 1 -- 172.21.15.60:0/871607709 shutdown_connections 2024-05-09T05:01:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.012+0000 7fb6ab42a640 1 --2- 172.21.15.60:0/871607709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb690004800 0x7fb690004c80 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.012+0000 7fb6ab42a640 1 -- 172.21.15.60:0/871607709 >> 172.21.15.60:0/871607709 conn(0x7fb6a4071e80 msgr2=0x7fb6a4072290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.012+0000 7fb6ab42a640 1 -- 172.21.15.60:0/871607709 shutdown_connections 2024-05-09T05:01:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.012+0000 7fb6ab42a640 1 -- 172.21.15.60:0/871607709 wait complete. 2024-05-09T05:01:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.013+0000 7fb6ab42a640 1 Processor -- start 2024-05-09T05:01:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.013+0000 7fb6ab42a640 1 -- start start 2024-05-09T05:01:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.014+0000 7fb6ab42a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb690004800 0x7fb6a4097be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.014+0000 7fb6ab42a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a406cd80 0x7fb6a41f5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.014+0000 7fb6ab42a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6a41f5580 0x7fb6a415acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.014+0000 7fb6ab42a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a40b3400 con 0x7fb690004800 2024-05-09T05:01:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.014+0000 7fb6ab42a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a40b3280 con 0x7fb6a406cd80 2024-05-09T05:01:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.014+0000 7fb6ab42a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb6a40b3580 con 0x7fb6a41f5580 2024-05-09T05:01:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.014+0000 7fb6a91c6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb690004800 0x7fb6a4097be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.014+0000 7fb6a91c6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb690004800 0x7fb6a4097be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37696/0 (socket says 172.21.15.60:37696) 2024-05-09T05:01:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.014+0000 7fb6a91c6640 1 -- 172.21.15.60:0/66603013 learned_addr learned my addr 172.21.15.60:0/66603013 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.014+0000 7fb6a91c6640 1 -- 172.21.15.60:0/66603013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6a41f5580 msgr2=0x7fb6a415acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.014+0000 7fb6a91c6640 1 --2- 172.21.15.60:0/66603013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6a41f5580 0x7fb6a415acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.014+0000 7fb6a89c5640 1 --2- 172.21.15.60:0/66603013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a406cd80 0x7fb6a41f5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.014+0000 7fb6a91c6640 1 -- 172.21.15.60:0/66603013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a406cd80 msgr2=0x7fb6a41f5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.014+0000 7fb6a91c6640 1 --2- 172.21.15.60:0/66603013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a406cd80 0x7fb6a41f5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.014+0000 7fb6a91c6640 1 -- 172.21.15.60:0/66603013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a4159ef0 con 0x7fb690004800 2024-05-09T05:01:43.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.015+0000 7fb6a91c6640 1 --2- 172.21.15.60:0/66603013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb690004800 0x7fb6a4097be0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fb6a000f770 tx=0x7fb6a000fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.015+0000 7fb699ffb640 1 -- 172.21.15.60:0/66603013 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a00050d0 con 0x7fb690004800 2024-05-09T05:01:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.015+0000 7fb699ffb640 1 -- 172.21.15.60:0/66603013 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6a0018070 con 0x7fb690004800 2024-05-09T05:01:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.015+0000 7fb6ab42a640 1 -- 172.21.15.60:0/66603013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6a415a1d0 con 0x7fb690004800 2024-05-09T05:01:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.015+0000 7fb699ffb640 1 -- 172.21.15.60:0/66603013 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a000d6b0 con 0x7fb690004800 2024-05-09T05:01:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.015+0000 7fb6ab42a640 1 -- 172.21.15.60:0/66603013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6a415a790 con 0x7fb690004800 2024-05-09T05:01:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.016+0000 7fb6ab42a640 1 -- 172.21.15.60:0/66603013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6a415a990 con 0x7fb690004800 2024-05-09T05:01:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.017+0000 7fb699ffb640 1 -- 172.21.15.60:0/66603013 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb6a000d850 con 0x7fb690004800 2024-05-09T05:01:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.017+0000 7fb699ffb640 1 --2- 172.21.15.60:0/66603013 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb688076300 0x7fb6880787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.017+0000 7fb699ffb640 1 -- 172.21.15.60:0/66603013 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fb6a00974d0 con 0x7fb690004800 2024-05-09T05:01:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.017+0000 7fb6a89c5640 1 --2- 172.21.15.60:0/66603013 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb688076300 0x7fb6880787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:43.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.018+0000 7fb6a89c5640 1 --2- 172.21.15.60:0/66603013 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb688076300 0x7fb6880787c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb694033000 tx=0x7fb694008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:43.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.021+0000 7fb699ffb640 1 -- 172.21.15.60:0/66603013 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb6a415a990 con 0x7fb690004800 2024-05-09T05:01:43.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.250+0000 7fb6ab42a640 1 -- 172.21.15.60:0/66603013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6a4051040 con 0x7fb690004800 2024-05-09T05:01:43.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.250+0000 7fb699ffb640 1 -- 172.21.15.60:0/66603013 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5187 (secure 0 0 0) 0x7fb6a0060d30 con 0x7fb690004800 2024-05-09T05:01:43.251 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:43.251 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3604145570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3604145570},{"type":"v1","addr":"172.21.15.60:6834","nonce":3604145570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3949194455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3949194455},{"type":"v1","addr":"172.21.15.114:6836","nonce":3949194455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3288630382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3288630382},{"type":"v1","addr":"172.21.15.114:6837","nonce":3288630382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:39.237498+0000","modified":"2024-05-09T05:01:42.253278+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5415":{"gid":5415,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2425263408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2425263408},{"type":"v1","addr":"172.21.15.60:6835","nonce":2425263408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5415,"qdb_cluster":[5415]},"id":3}]} 2024-05-09T05:01:43.252 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 27 2024-05-09T05:01:43.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.252+0000 7fb6ab42a640 1 -- 172.21.15.60:0/66603013 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb688076300 msgr2=0x7fb6880787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.252+0000 7fb6ab42a640 1 --2- 172.21.15.60:0/66603013 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb688076300 0x7fb6880787c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb694033000 tx=0x7fb694008030 comp rx=0 tx=0).stop 2024-05-09T05:01:43.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.252+0000 7fb6ab42a640 1 -- 172.21.15.60:0/66603013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb690004800 msgr2=0x7fb6a4097be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.252+0000 7fb6ab42a640 1 --2- 172.21.15.60:0/66603013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb690004800 0x7fb6a4097be0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fb6a000f770 tx=0x7fb6a000fc30 comp rx=0 tx=0).stop 2024-05-09T05:01:43.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.252+0000 7fb6ab42a640 1 -- 172.21.15.60:0/66603013 shutdown_connections 2024-05-09T05:01:43.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.252+0000 7fb6ab42a640 1 --2- 172.21.15.60:0/66603013 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb688076300 0x7fb6880787c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.252+0000 7fb6ab42a640 1 --2- 172.21.15.60:0/66603013 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6a41f5580 0x7fb6a415acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.252+0000 7fb6ab42a640 1 --2- 172.21.15.60:0/66603013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6a406cd80 0x7fb6a41f5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.252+0000 7fb6ab42a640 1 --2- 172.21.15.60:0/66603013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb690004800 0x7fb6a4097be0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.252+0000 7fb6ab42a640 1 -- 172.21.15.60:0/66603013 >> 172.21.15.60:0/66603013 conn(0x7fb6a4071e80 msgr2=0x7fb6a40ae910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:43.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.252+0000 7fb6ab42a640 1 -- 172.21.15.60:0/66603013 shutdown_connections 2024-05-09T05:01:43.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.252+0000 7fb6ab42a640 1 -- 172.21.15.60:0/66603013 wait complete. 2024-05-09T05:01:43.264 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:01:43.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.367+0000 7f54a61e7640 1 Processor -- start 2024-05-09T05:01:43.368 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.367+0000 7f54a61e7640 1 -- start start 2024-05-09T05:01:43.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.368+0000 7f54a61e7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a015c3a0 0x7f54a0070150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:43.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.368+0000 7f54a61e7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f54a00ac430 0x7f54a0070690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:43.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.368+0000 7f54a61e7640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54a0071b80 con 0x7f54a0097de0 2024-05-09T05:01:43.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.368+0000 7f54a61e7640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f54a0071e60 con 0x7f54a00ac430 2024-05-09T05:01:43.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.368+0000 7f54a61e7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f54a0072140 con 0x7f54a015c3a0 2024-05-09T05:01:43.369 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.368+0000 7f549effd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f54a00ac430 0x7f54a0070690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:43.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.368+0000 7f549effd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f54a00ac430 0x7f54a0070690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37698/0 (socket says 172.21.15.60:37698) 2024-05-09T05:01:43.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.368+0000 7f549effd640 1 -- 172.21.15.60:0/2466226130 learned_addr learned my addr 172.21.15.60:0/2466226130 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:43.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.368+0000 7f549f7fe640 1 --2- 172.21.15.60:0/2466226130 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a015c3a0 0x7f54a0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:43.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.368+0000 7f549effd640 1 -- 172.21.15.60:0/2466226130 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a015c3a0 msgr2=0x7f54a0070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:43.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.369+0000 7f549effd640 1 --2- 172.21.15.60:0/2466226130 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a015c3a0 0x7f54a0070150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.369+0000 7f549effd640 1 -- 172.21.15.60:0/2466226130 >> v1:172.21.15.114:6789/0 conn(0x7f54a0097de0 legacy=0x7f54a0099a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.369+0000 7f549effd640 1 -- 172.21.15.60:0/2466226130 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a0072420 con 0x7f54a00ac430 2024-05-09T05:01:43.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.369+0000 7f549f7fe640 1 --2- 172.21.15.60:0/2466226130 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a015c3a0 0x7f54a0070150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:43.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.369+0000 7f549effd640 1 --2- 172.21.15.60:0/2466226130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54a00ac430 0x7f54a0070690 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f54940098a0 tx=0x7f549402a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d9cf3efc03367f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:43.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.369+0000 7f549e7fc640 1 -- 172.21.15.60:0/2466226130 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5494037070 con 0x7f54a00ac430 2024-05-09T05:01:43.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.369+0000 7f549e7fc640 1 -- 172.21.15.60:0/2466226130 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5494032440 con 0x7f54a00ac430 2024-05-09T05:01:43.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.369+0000 7f549e7fc640 1 -- 172.21.15.60:0/2466226130 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5494032810 con 0x7f54a00ac430 2024-05-09T05:01:43.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.370+0000 7f54a61e7640 1 -- 172.21.15.60:0/2466226130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54a00ac430 msgr2=0x7f54a0070690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.370+0000 7f54a61e7640 1 --2- 172.21.15.60:0/2466226130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54a00ac430 0x7f54a0070690 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f54940098a0 tx=0x7f549402a290 comp rx=0 tx=0).stop 2024-05-09T05:01:43.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.370+0000 7f54a61e7640 1 -- 172.21.15.60:0/2466226130 shutdown_connections 2024-05-09T05:01:43.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.370+0000 7f54a61e7640 1 --2- 172.21.15.60:0/2466226130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54a00ac430 0x7f54a0070690 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.370+0000 7f54a61e7640 1 --2- 172.21.15.60:0/2466226130 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a015c3a0 0x7f54a0070150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.370+0000 7f54a61e7640 1 -- 172.21.15.60:0/2466226130 >> 172.21.15.60:0/2466226130 conn(0x7f54a00972f0 msgr2=0x7f54a00976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:43.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.370+0000 7f54a61e7640 1 -- 172.21.15.60:0/2466226130 shutdown_connections 2024-05-09T05:01:43.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.370+0000 7f54a61e7640 1 -- 172.21.15.60:0/2466226130 wait complete. 2024-05-09T05:01:43.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.371+0000 7f54a61e7640 1 Processor -- start 2024-05-09T05:01:43.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.371+0000 7f54a61e7640 1 -- start start 2024-05-09T05:01:43.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.372+0000 7f54a61e7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54a0097de0 0x7f54a00a9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:43.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.372+0000 7f54a61e7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a00ac430 0x7f54a0155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:43.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.372+0000 7f54a61e7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54a015c3a0 0x7f54a0155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:43.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.372+0000 7f54a61e7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f54a0071810 con 0x7f54a0097de0 2024-05-09T05:01:43.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.372+0000 7f54a61e7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f54a015c780 con 0x7f54a015c3a0 2024-05-09T05:01:43.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.372+0000 7f54a61e7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f54a0073c30 con 0x7f54a00ac430 2024-05-09T05:01:43.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.372+0000 7f549f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54a0097de0 0x7f54a00a9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:43.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.372+0000 7f549f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54a0097de0 0x7f54a00a9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37712/0 (socket says 172.21.15.60:37712) 2024-05-09T05:01:43.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.372+0000 7f549f7fe640 1 -- 172.21.15.60:0/542886793 learned_addr learned my addr 172.21.15.60:0/542886793 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:43.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.372+0000 7f549effd640 1 --2- 172.21.15.60:0/542886793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a00ac430 0x7f54a0155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:43.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.372+0000 7f549ffff640 1 --2- 172.21.15.60:0/542886793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54a015c3a0 0x7f54a0155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:43.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.373+0000 7f549f7fe640 1 -- 172.21.15.60:0/542886793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a00ac430 msgr2=0x7f54a0155680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.373+0000 7f549f7fe640 1 --2- 172.21.15.60:0/542886793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a00ac430 0x7f54a0155680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.373+0000 7f549f7fe640 1 -- 172.21.15.60:0/542886793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54a015c3a0 msgr2=0x7f54a0155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.373+0000 7f549f7fe640 1 --2- 172.21.15.60:0/542886793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54a015c3a0 0x7f54a0155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.373+0000 7f549f7fe640 1 -- 172.21.15.60:0/542886793 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a0096f40 con 0x7f54a0097de0 2024-05-09T05:01:43.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.373+0000 7f549effd640 1 --2- 172.21.15.60:0/542886793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a00ac430 0x7f54a0155680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:43.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.373+0000 7f549ffff640 1 --2- 172.21.15.60:0/542886793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54a015c3a0 0x7f54a0155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:43.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.373+0000 7f549f7fe640 1 --2- 172.21.15.60:0/542886793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54a0097de0 0x7f54a00a9350 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f549000d800 tx=0x7f549000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:43.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.373+0000 7f5487fff640 1 -- 172.21.15.60:0/542886793 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5490004030 con 0x7f54a0097de0 2024-05-09T05:01:43.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.373+0000 7f5487fff640 1 -- 172.21.15.60:0/542886793 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54900041d0 con 0x7f54a0097de0 2024-05-09T05:01:43.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.373+0000 7f54a61e7640 1 -- 172.21.15.60:0/542886793 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54a0096980 con 0x7f54a0097de0 2024-05-09T05:01:43.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.373+0000 7f5487fff640 1 -- 172.21.15.60:0/542886793 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54900096a0 con 0x7f54a0097de0 2024-05-09T05:01:43.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.373+0000 7f54a61e7640 1 -- 172.21.15.60:0/542886793 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54a00a9890 con 0x7f54a0097de0 2024-05-09T05:01:43.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.374+0000 7f54a61e7640 1 -- 172.21.15.60:0/542886793 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54a0098e00 con 0x7f54a0097de0 2024-05-09T05:01:43.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.375+0000 7f5487fff640 1 -- 172.21.15.60:0/542886793 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5490009840 con 0x7f54a0097de0 2024-05-09T05:01:43.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.375+0000 7f5487fff640 1 --2- 172.21.15.60:0/542886793 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f548c0762d0 0x7f548c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:43.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.375+0000 7f5487fff640 1 -- 172.21.15.60:0/542886793 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f54900979e0 con 0x7f54a0097de0 2024-05-09T05:01:43.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.375+0000 7f549effd640 1 --2- 172.21.15.60:0/542886793 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f548c0762d0 0x7f548c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:43.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.376+0000 7f549effd640 1 --2- 172.21.15.60:0/542886793 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f548c0762d0 0x7f548c078790 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5494034000 tx=0x7f549402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:43.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.379+0000 7f5487fff640 1 -- 172.21.15.60:0/542886793 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f54900611b0 con 0x7f54a0097de0 2024-05-09T05:01:43.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.570+0000 7f54a61e7640 1 -- 172.21.15.60:0/542886793 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f54a0051040 con 0x7f54a0097de0 2024-05-09T05:01:43.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.571+0000 7f5487fff640 1 -- 172.21.15.60:0/542886793 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) ==== 74+0+16914 (secure 0 0 0) 0x7f5490060ab0 con 0x7f54a0097de0 2024-05-09T05:01:43.572 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:43.572 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":51,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:01:40.847633+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:01:36.910324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T05:01:38.185514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:01:43.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.573+0000 7f54a61e7640 1 -- 172.21.15.60:0/542886793 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f548c0762d0 msgr2=0x7f548c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.573+0000 7f54a61e7640 1 --2- 172.21.15.60:0/542886793 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f548c0762d0 0x7f548c078790 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5494034000 tx=0x7f549402e000 comp rx=0 tx=0).stop 2024-05-09T05:01:43.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.573+0000 7f54a61e7640 1 -- 172.21.15.60:0/542886793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54a0097de0 msgr2=0x7f54a00a9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.573+0000 7f54a61e7640 1 --2- 172.21.15.60:0/542886793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54a0097de0 0x7f54a00a9350 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f549000d800 tx=0x7f549000dcc0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.573+0000 7f54a61e7640 1 -- 172.21.15.60:0/542886793 shutdown_connections 2024-05-09T05:01:43.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.573+0000 7f54a61e7640 1 --2- 172.21.15.60:0/542886793 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f548c0762d0 0x7f548c078790 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.573+0000 7f54a61e7640 1 --2- 172.21.15.60:0/542886793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54a015c3a0 0x7f54a0155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.573+0000 7f54a61e7640 1 --2- 172.21.15.60:0/542886793 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f54a00ac430 0x7f54a0155680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.573+0000 7f54a61e7640 1 --2- 172.21.15.60:0/542886793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54a0097de0 0x7f54a00a9350 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.573+0000 7f54a61e7640 1 -- 172.21.15.60:0/542886793 >> 172.21.15.60:0/542886793 conn(0x7f54a00972f0 msgr2=0x7f54a0099a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:43.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.573+0000 7f54a61e7640 1 -- 172.21.15.60:0/542886793 shutdown_connections 2024-05-09T05:01:43.575 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.573+0000 7f54a61e7640 1 -- 172.21.15.60:0/542886793 wait complete. 2024-05-09T05:01:43.585 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-09T05:01:43.585 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:01:43.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.689+0000 7f6404bd0640 1 Processor -- start 2024-05-09T05:01:43.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.689+0000 7f6404bd0640 1 -- start start 2024-05-09T05:01:43.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.690+0000 7f6404bd0640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f640006f9f0 0x7f640006ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:43.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.690+0000 7f6404bd0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f640006ff60 con 0x7f64001606f0 2024-05-09T05:01:43.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.690+0000 7f6404bd0640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6400070240 con 0x7f640015f300 2024-05-09T05:01:43.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.690+0000 7f6404bd0640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f6400070520 con 0x7f640006f9f0 2024-05-09T05:01:43.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.690+0000 7f63f7fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f64001606f0 0x7f64000d33f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57756/0 (socket says 172.21.15.60:57756) 2024-05-09T05:01:43.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.690+0000 7f63f7fff640 1 -- 172.21.15.60:0/619624242 learned_addr learned my addr 172.21.15.60:0/619624242 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:43.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.690+0000 7f63fed9d640 1 --2- 172.21.15.60:0/619624242 >> v2:172.21.15.114:3300/0 conn(0x7f640006f9f0 0x7f640006ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:43.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.690+0000 7f63fdd9b640 1 -- 172.21.15.60:0/619624242 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4150647342 0 0) 0x7f640006ff60 con 0x7f64001606f0 2024-05-09T05:01:43.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.690+0000 7f63fdd9b640 1 -- 172.21.15.60:0/619624242 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63ec003520 con 0x7f64001606f0 2024-05-09T05:01:43.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.690+0000 7f63fed9d640 1 -- 172.21.15.60:0/619624242 >> v1:172.21.15.114:6789/0 conn(0x7f640015f300 legacy=0x7f64001639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.690+0000 7f63fdd9b640 1 -- 172.21.15.60:0/619624242 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4290512095 0 0) 0x7f6400070240 con 0x7f640015f300 2024-05-09T05:01:43.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.690+0000 7f63fed9d640 1 -- 172.21.15.60:0/619624242 >> v1:172.21.15.60:6789/0 conn(0x7f64001606f0 legacy=0x7f64000d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.690+0000 7f63fed9d640 1 -- 172.21.15.60:0/619624242 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f640006e740 con 0x7f640006f9f0 2024-05-09T05:01:43.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.691+0000 7f63fed9d640 1 --2- 172.21.15.60:0/619624242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f640006f9f0 0x7f640006ef40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009dc0 tx=0x7f63f802a6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d098a4e7688587e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:43.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.691+0000 7f63fdd9b640 1 -- 172.21.15.60:0/619624242 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f8035020 con 0x7f640006f9f0 2024-05-09T05:01:43.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.691+0000 7f63fdd9b640 1 -- 172.21.15.60:0/619624242 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63f8007540 con 0x7f640006f9f0 2024-05-09T05:01:43.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.691+0000 7f63fdd9b640 1 -- 172.21.15.60:0/619624242 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e80016f0 con 0x7f640006f9f0 2024-05-09T05:01:43.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.692+0000 7f6404bd0640 1 -- 172.21.15.60:0/619624242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f640006f9f0 msgr2=0x7f640006ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.692+0000 7f6404bd0640 1 --2- 172.21.15.60:0/619624242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f640006f9f0 0x7f640006ef40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009dc0 tx=0x7f63f802a6a0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.692+0000 7f6404bd0640 1 -- 172.21.15.60:0/619624242 shutdown_connections 2024-05-09T05:01:43.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.692+0000 7f6404bd0640 1 --2- 172.21.15.60:0/619624242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f640006f9f0 0x7f640006ef40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.692+0000 7f6404bd0640 1 -- 172.21.15.60:0/619624242 >> 172.21.15.60:0/619624242 conn(0x7f640009b850 msgr2=0x7f640009bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:43.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.692+0000 7f6404bd0640 1 -- 172.21.15.60:0/619624242 shutdown_connections 2024-05-09T05:01:43.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.692+0000 7f6404bd0640 1 -- 172.21.15.60:0/619624242 wait complete. 2024-05-09T05:01:43.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.693+0000 7f6404bd0640 1 Processor -- start 2024-05-09T05:01:43.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.693+0000 7f6404bd0640 1 -- start start 2024-05-09T05:01:43.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.694+0000 7f6404bd0640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f640006f9f0 0x7f640015bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:43.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.694+0000 7f6404bd0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f640015f300 0x7f64001f5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:43.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.694+0000 7f6404bd0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64001606f0 0x7f64001f62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:43.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.694+0000 7f6404bd0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f64000d3c80 con 0x7f640015f300 2024-05-09T05:01:43.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.694+0000 7f6404bd0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f64000d3b00 con 0x7f64001606f0 2024-05-09T05:01:43.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.694+0000 7f6404bd0640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f640006f480 con 0x7f640006f9f0 2024-05-09T05:01:43.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.694+0000 7f63f7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f640015f300 0x7f64001f5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:43.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.694+0000 7f63f7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f640015f300 0x7f64001f5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37716/0 (socket says 172.21.15.60:37716) 2024-05-09T05:01:43.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.694+0000 7f63f7fff640 1 -- 172.21.15.60:0/940144151 learned_addr learned my addr 172.21.15.60:0/940144151 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:43.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.694+0000 7f63fe59c640 1 --2- 172.21.15.60:0/940144151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f640006f9f0 0x7f640015bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:43.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.694+0000 7f63f7fff640 1 -- 172.21.15.60:0/940144151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f640006f9f0 msgr2=0x7f640015bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.694+0000 7f63fed9d640 1 --2- 172.21.15.60:0/940144151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64001606f0 0x7f64001f62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:43.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.694+0000 7f63f7fff640 1 --2- 172.21.15.60:0/940144151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f640006f9f0 0x7f640015bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.694+0000 7f63f7fff640 1 -- 172.21.15.60:0/940144151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64001606f0 msgr2=0x7f64001f62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.694+0000 7f63f7fff640 1 --2- 172.21.15.60:0/940144151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64001606f0 0x7f64001f62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.694+0000 7f63f7fff640 1 -- 172.21.15.60:0/940144151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64001f60b0 con 0x7f640015f300 2024-05-09T05:01:43.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.695+0000 7f63f7fff640 1 --2- 172.21.15.60:0/940144151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f640015f300 0x7f64001f5b70 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f63e800c700 tx=0x7f63e800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:43.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.695+0000 7f63f6ffd640 1 -- 172.21.15.60:0/940144151 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e8005ad0 con 0x7f640015f300 2024-05-09T05:01:43.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.695+0000 7f63f6ffd640 1 -- 172.21.15.60:0/940144151 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63e8005c70 con 0x7f640015f300 2024-05-09T05:01:43.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.695+0000 7f6404bd0640 1 -- 172.21.15.60:0/940144151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f640009d960 con 0x7f640015f300 2024-05-09T05:01:43.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.695+0000 7f63f6ffd640 1 -- 172.21.15.60:0/940144151 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e8007290 con 0x7f640015f300 2024-05-09T05:01:43.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.695+0000 7f6404bd0640 1 -- 172.21.15.60:0/940144151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f640009dc00 con 0x7f640015f300 2024-05-09T05:01:43.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.696+0000 7f6404bd0640 1 -- 172.21.15.60:0/940144151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f640009bee0 con 0x7f640015f300 2024-05-09T05:01:43.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.696+0000 7f63f6ffd640 1 -- 172.21.15.60:0/940144151 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f63e8007430 con 0x7f640015f300 2024-05-09T05:01:43.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.697+0000 7f63f6ffd640 1 --2- 172.21.15.60:0/940144151 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f63e00762d0 0x7f63e0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:43.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.697+0000 7f63f6ffd640 1 -- 172.21.15.60:0/940144151 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f63e80975a0 con 0x7f640015f300 2024-05-09T05:01:43.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.697+0000 7f63fe59c640 1 --2- 172.21.15.60:0/940144151 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f63e00762d0 0x7f63e0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:43.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.698+0000 7f63fe59c640 1 --2- 172.21.15.60:0/940144151 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f63e00762d0 0x7f63e0078790 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f63f0001220 tx=0x7f63f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:43.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.702+0000 7f63f6ffd640 1 -- 172.21.15.60:0/940144151 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f640009bee0 con 0x7f640015f300 2024-05-09T05:01:43.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.889+0000 7f6404bd0640 1 -- 172.21.15.60:0/940144151 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f640015c100 con 0x7f63e00762d0 2024-05-09T05:01:43.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.896+0000 7f63f6ffd640 1 -- 172.21.15.60:0/940144151 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251942 (secure 0 0 0) 0x7f640015c100 con 0x7f63e00762d0 2024-05-09T05:01:43.899 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:43.901 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-05-09T05:01:43.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.899+0000 7f6404bd0640 1 -- 172.21.15.60:0/940144151 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f63e00762d0 msgr2=0x7f63e0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.899+0000 7f6404bd0640 1 --2- 172.21.15.60:0/940144151 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f63e00762d0 0x7f63e0078790 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f63f0001220 tx=0x7f63f0009040 comp rx=0 tx=0).stop 2024-05-09T05:01:43.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.899+0000 7f6404bd0640 1 -- 172.21.15.60:0/940144151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f640015f300 msgr2=0x7f64001f5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:43.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.899+0000 7f6404bd0640 1 --2- 172.21.15.60:0/940144151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f640015f300 0x7f64001f5b70 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f63e800c700 tx=0x7f63e800cbc0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.899+0000 7f6404bd0640 1 -- 172.21.15.60:0/940144151 shutdown_connections 2024-05-09T05:01:43.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.899+0000 7f6404bd0640 1 --2- 172.21.15.60:0/940144151 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f63e00762d0 0x7f63e0078790 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.899+0000 7f6404bd0640 1 --2- 172.21.15.60:0/940144151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64001606f0 0x7f64001f62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.899+0000 7f6404bd0640 1 --2- 172.21.15.60:0/940144151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f640015f300 0x7f64001f5b70 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.899+0000 7f6404bd0640 1 --2- 172.21.15.60:0/940144151 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f640006f9f0 0x7f640015bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:43.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.899+0000 7f6404bd0640 1 -- 172.21.15.60:0/940144151 >> 172.21.15.60:0/940144151 conn(0x7f640009b850 msgr2=0x7f640009a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:43.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.899+0000 7f6404bd0640 1 -- 172.21.15.60:0/940144151 shutdown_connections 2024-05-09T05:01:43.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:43.899+0000 7f6404bd0640 1 -- 172.21.15.60:0/940144151 wait complete. 2024-05-09T05:01:43.911 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-05-09T05:01:42.930309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238792,"kb_used_data":5968,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735928,"statfs":{"total":773094113280,"available":772849590272,"internally_reserved":0,"allocated":6111232,"data_stored":3341700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.084541"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.852002+0000","last_change":"2024-05-09T05:01:39.875218+0000","last_active":"2024-05-09T05:01:40.852002+0000","last_peered":"2024-05-09T05:01:40.852002+0000","last_clean":"2024-05-09T05:01:40.852002+0000","last_became_active":"2024-05-09T05:01:39.874925+0000","last_became_peered":"2024-05-09T05:01:39.874925+0000","last_unstale":"2024-05-09T05:01:40.852002+0000","last_undegraded":"2024-05-09T05:01:40.852002+0000","last_fullsized":"2024-05-09T05:01:40.852002+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:12.625336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851053+0000","last_change":"2024-05-09T05:01:39.864266+0000","last_active":"2024-05-09T05:01:40.851053+0000","last_peered":"2024-05-09T05:01:40.851053+0000","last_clean":"2024-05-09T05:01:40.851053+0000","last_became_active":"2024-05-09T05:01:39.863965+0000","last_became_peered":"2024-05-09T05:01:39.863965+0000","last_unstale":"2024-05-09T05:01:40.851053+0000","last_undegraded":"2024-05-09T05:01:40.851053+0000","last_fullsized":"2024-05-09T05:01:40.851053+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:01:57.964834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857444+0000","last_change":"2024-05-09T05:01:39.867479+0000","last_active":"2024-05-09T05:01:40.857444+0000","last_peered":"2024-05-09T05:01:40.857444+0000","last_clean":"2024-05-09T05:01:40.857444+0000","last_became_active":"2024-05-09T05:01:39.867215+0000","last_became_peered":"2024-05-09T05:01:39.867215+0000","last_unstale":"2024-05-09T05:01:40.857444+0000","last_undegraded":"2024-05-09T05:01:40.857444+0000","last_fullsized":"2024-05-09T05:01:40.857444+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:57:48.604326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358390+0000","last_change":"2024-05-09T05:01:39.864540+0000","last_active":"2024-05-09T05:01:41.358390+0000","last_peered":"2024-05-09T05:01:41.358390+0000","last_clean":"2024-05-09T05:01:41.358390+0000","last_became_active":"2024-05-09T05:01:39.864203+0000","last_became_peered":"2024-05-09T05:01:39.864203+0000","last_unstale":"2024-05-09T05:01:41.358390+0000","last_undegraded":"2024-05-09T05:01:41.358390+0000","last_fullsized":"2024-05-09T05:01:41.358390+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:48:03.359391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851847+0000","last_change":"2024-05-09T05:01:39.871306+0000","last_active":"2024-05-09T05:01:40.851847+0000","last_peered":"2024-05-09T05:01:40.851847+0000","last_clean":"2024-05-09T05:01:40.851847+0000","last_became_active":"2024-05-09T05:01:39.870998+0000","last_became_peered":"2024-05-09T05:01:39.870998+0000","last_unstale":"2024-05-09T05:01:40.851847+0000","last_undegraded":"2024-05-09T05:01:40.851847+0000","last_fullsized":"2024-05-09T05:01:40.851847+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:38.550789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358285+0000","last_change":"2024-05-09T05:01:39.875924+0000","last_active":"2024-05-09T05:01:41.358285+0000","last_peered":"2024-05-09T05:01:41.358285+0000","last_clean":"2024-05-09T05:01:41.358285+0000","last_became_active":"2024-05-09T05:01:39.875178+0000","last_became_peered":"2024-05-09T05:01:39.875178+0000","last_unstale":"2024-05-09T05:01:41.358285+0000","last_undegraded":"2024-05-09T05:01:41.358285+0000","last_fullsized":"2024-05-09T05:01:41.358285+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:37:52.332218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980355+0000","last_change":"2024-05-09T05:01:39.870489+0000","last_active":"2024-05-09T05:01:40.980355+0000","last_peered":"2024-05-09T05:01:40.980355+0000","last_clean":"2024-05-09T05:01:40.980355+0000","last_became_active":"2024-05-09T05:01:39.868433+0000","last_became_peered":"2024-05-09T05:01:39.868433+0000","last_unstale":"2024-05-09T05:01:40.980355+0000","last_undegraded":"2024-05-09T05:01:40.980355+0000","last_fullsized":"2024-05-09T05:01:40.980355+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:06.258896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.850921+0000","last_change":"2024-05-09T05:01:39.854222+0000","last_active":"2024-05-09T05:01:40.850921+0000","last_peered":"2024-05-09T05:01:40.850921+0000","last_clean":"2024-05-09T05:01:40.850921+0000","last_became_active":"2024-05-09T05:01:39.853927+0000","last_became_peered":"2024-05-09T05:01:39.853927+0000","last_unstale":"2024-05-09T05:01:40.850921+0000","last_undegraded":"2024-05-09T05:01:40.850921+0000","last_fullsized":"2024-05-09T05:01:40.850921+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:41:31.058309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980674+0000","last_change":"2024-05-09T05:01:39.861561+0000","last_active":"2024-05-09T05:01:40.980674+0000","last_peered":"2024-05-09T05:01:40.980674+0000","last_clean":"2024-05-09T05:01:40.980674+0000","last_became_active":"2024-05-09T05:01:39.861026+0000","last_became_peered":"2024-05-09T05:01:39.861026+0000","last_unstale":"2024-05-09T05:01:40.980674+0000","last_undegraded":"2024-05-09T05:01:40.980674+0000","last_fullsized":"2024-05-09T05:01:40.980674+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:16:28.592034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358930+0000","last_change":"2024-05-09T05:01:39.866566+0000","last_active":"2024-05-09T05:01:41.358930+0000","last_peered":"2024-05-09T05:01:41.358930+0000","last_clean":"2024-05-09T05:01:41.358930+0000","last_became_active":"2024-05-09T05:01:39.866315+0000","last_became_peered":"2024-05-09T05:01:39.866315+0000","last_unstale":"2024-05-09T05:01:41.358930+0000","last_undegraded":"2024-05-09T05:01:41.358930+0000","last_fullsized":"2024-05-09T05:01:41.358930+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:05:22.823683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358777+0000","last_change":"2024-05-09T05:01:39.869309+0000","last_active":"2024-05-09T05:01:41.358777+0000","last_peered":"2024-05-09T05:01:41.358777+0000","last_clean":"2024-05-09T05:01:41.358777+0000","last_became_active":"2024-05-09T05:01:39.868934+0000","last_became_peered":"2024-05-09T05:01:39.868934+0000","last_unstale":"2024-05-09T05:01:41.358777+0000","last_undegraded":"2024-05-09T05:01:41.358777+0000","last_fullsized":"2024-05-09T05:01:41.358777+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:47:16.952818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358583+0000","last_change":"2024-05-09T05:01:39.873199+0000","last_active":"2024-05-09T05:01:41.358583+0000","last_peered":"2024-05-09T05:01:41.358583+0000","last_clean":"2024-05-09T05:01:41.358583+0000","last_became_active":"2024-05-09T05:01:39.872903+0000","last_became_peered":"2024-05-09T05:01:39.872903+0000","last_unstale":"2024-05-09T05:01:41.358583+0000","last_undegraded":"2024-05-09T05:01:41.358583+0000","last_fullsized":"2024-05-09T05:01:41.358583+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:17:38.783370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358286+0000","last_change":"2024-05-09T05:01:39.874859+0000","last_active":"2024-05-09T05:01:41.358286+0000","last_peered":"2024-05-09T05:01:41.358286+0000","last_clean":"2024-05-09T05:01:41.358286+0000","last_became_active":"2024-05-09T05:01:39.874533+0000","last_became_peered":"2024-05-09T05:01:39.874533+0000","last_unstale":"2024-05-09T05:01:41.358286+0000","last_undegraded":"2024-05-09T05:01:41.358286+0000","last_fullsized":"2024-05-09T05:01:41.358286+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:35.258317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980624+0000","last_change":"2024-05-09T05:01:39.865170+0000","last_active":"2024-05-09T05:01:40.980624+0000","last_peered":"2024-05-09T05:01:40.980624+0000","last_clean":"2024-05-09T05:01:40.980624+0000","last_became_active":"2024-05-09T05:01:39.864867+0000","last_became_peered":"2024-05-09T05:01:39.864867+0000","last_unstale":"2024-05-09T05:01:40.980624+0000","last_undegraded":"2024-05-09T05:01:40.980624+0000","last_fullsized":"2024-05-09T05:01:40.980624+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:21:29.020025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980606+0000","last_change":"2024-05-09T05:01:39.868574+0000","last_active":"2024-05-09T05:01:40.980606+0000","last_peered":"2024-05-09T05:01:40.980606+0000","last_clean":"2024-05-09T05:01:40.980606+0000","last_became_active":"2024-05-09T05:01:39.865956+0000","last_became_peered":"2024-05-09T05:01:39.865956+0000","last_unstale":"2024-05-09T05:01:40.980606+0000","last_undegraded":"2024-05-09T05:01:40.980606+0000","last_fullsized":"2024-05-09T05:01:40.980606+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:27:07.564385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358004+0000","last_change":"2024-05-09T05:01:39.867547+0000","last_active":"2024-05-09T05:01:41.358004+0000","last_peered":"2024-05-09T05:01:41.358004+0000","last_clean":"2024-05-09T05:01:41.358004+0000","last_became_active":"2024-05-09T05:01:39.867266+0000","last_became_peered":"2024-05-09T05:01:39.867266+0000","last_unstale":"2024-05-09T05:01:41.358004+0000","last_undegraded":"2024-05-09T05:01:41.358004+0000","last_fullsized":"2024-05-09T05:01:41.358004+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:24.553056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851453+0000","last_change":"2024-05-09T05:01:39.872382+0000","last_active":"2024-05-09T05:01:40.851453+0000","last_peered":"2024-05-09T05:01:40.851453+0000","last_clean":"2024-05-09T05:01:40.851453+0000","last_became_active":"2024-05-09T05:01:39.872055+0000","last_became_peered":"2024-05-09T05:01:39.872055+0000","last_unstale":"2024-05-09T05:01:40.851453+0000","last_undegraded":"2024-05-09T05:01:40.851453+0000","last_fullsized":"2024-05-09T05:01:40.851453+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:34:58.245332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857667+0000","last_change":"2024-05-09T05:01:39.863769+0000","last_active":"2024-05-09T05:01:40.857667+0000","last_peered":"2024-05-09T05:01:40.857667+0000","last_clean":"2024-05-09T05:01:40.857667+0000","last_became_active":"2024-05-09T05:01:39.863396+0000","last_became_peered":"2024-05-09T05:01:39.863396+0000","last_unstale":"2024-05-09T05:01:40.857667+0000","last_undegraded":"2024-05-09T05:01:40.857667+0000","last_fullsized":"2024-05-09T05:01:40.857667+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:02:53.110445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851562+0000","last_change":"2024-05-09T05:01:39.875936+0000","last_active":"2024-05-09T05:01:40.851562+0000","last_peered":"2024-05-09T05:01:40.851562+0000","last_clean":"2024-05-09T05:01:40.851562+0000","last_became_active":"2024-05-09T05:01:39.875614+0000","last_became_peered":"2024-05-09T05:01:39.875614+0000","last_unstale":"2024-05-09T05:01:40.851562+0000","last_undegraded":"2024-05-09T05:01:40.851562+0000","last_fullsized":"2024-05-09T05:01:40.851562+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:30.745037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851207+0000","last_change":"2024-05-09T05:01:39.867227+0000","last_active":"2024-05-09T05:01:40.851207+0000","last_peered":"2024-05-09T05:01:40.851207+0000","last_clean":"2024-05-09T05:01:40.851207+0000","last_became_active":"2024-05-09T05:01:39.866882+0000","last_became_peered":"2024-05-09T05:01:39.866882+0000","last_unstale":"2024-05-09T05:01:40.851207+0000","last_undegraded":"2024-05-09T05:01:40.851207+0000","last_fullsized":"2024-05-09T05:01:40.851207+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:01:16.557587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857353+0000","last_change":"2024-05-09T05:01:39.872958+0000","last_active":"2024-05-09T05:01:40.857353+0000","last_peered":"2024-05-09T05:01:40.857353+0000","last_clean":"2024-05-09T05:01:40.857353+0000","last_became_active":"2024-05-09T05:01:39.872567+0000","last_became_peered":"2024-05-09T05:01:39.872567+0000","last_unstale":"2024-05-09T05:01:40.857353+0000","last_undegraded":"2024-05-09T05:01:40.857353+0000","last_fullsized":"2024-05-09T05:01:40.857353+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:21.336536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851512+0000","last_change":"2024-05-09T05:01:39.873730+0000","last_active":"2024-05-09T05:01:40.851512+0000","last_peered":"2024-05-09T05:01:40.851512+0000","last_clean":"2024-05-09T05:01:40.851512+0000","last_became_active":"2024-05-09T05:01:39.873271+0000","last_became_peered":"2024-05-09T05:01:39.873271+0000","last_unstale":"2024-05-09T05:01:40.851512+0000","last_undegraded":"2024-05-09T05:01:40.851512+0000","last_fullsized":"2024-05-09T05:01:40.851512+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:51:42.545363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857303+0000","last_change":"2024-05-09T05:01:39.873510+0000","last_active":"2024-05-09T05:01:40.857303+0000","last_peered":"2024-05-09T05:01:40.857303+0000","last_clean":"2024-05-09T05:01:40.857303+0000","last_became_active":"2024-05-09T05:01:39.873226+0000","last_became_peered":"2024-05-09T05:01:39.873226+0000","last_unstale":"2024-05-09T05:01:40.857303+0000","last_undegraded":"2024-05-09T05:01:40.857303+0000","last_fullsized":"2024-05-09T05:01:40.857303+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:37:52.332218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.357859+0000","last_change":"2024-05-09T05:01:39.866948+0000","last_active":"2024-05-09T05:01:41.357859+0000","last_peered":"2024-05-09T05:01:41.357859+0000","last_clean":"2024-05-09T05:01:41.357859+0000","last_became_active":"2024-05-09T05:01:39.866329+0000","last_became_peered":"2024-05-09T05:01:39.866329+0000","last_unstale":"2024-05-09T05:01:41.357859+0000","last_undegraded":"2024-05-09T05:01:41.357859+0000","last_fullsized":"2024-05-09T05:01:41.357859+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:06.258896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.852139+0000","last_change":"2024-05-09T05:01:39.880371+0000","last_active":"2024-05-09T05:01:40.852139+0000","last_peered":"2024-05-09T05:01:40.852139+0000","last_clean":"2024-05-09T05:01:40.852139+0000","last_became_active":"2024-05-09T05:01:39.880114+0000","last_became_peered":"2024-05-09T05:01:39.880114+0000","last_unstale":"2024-05-09T05:01:40.852139+0000","last_undegraded":"2024-05-09T05:01:40.852139+0000","last_fullsized":"2024-05-09T05:01:40.852139+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:07:02.245459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358343+0000","last_change":"2024-05-09T05:01:39.871088+0000","last_active":"2024-05-09T05:01:41.358343+0000","last_peered":"2024-05-09T05:01:41.358343+0000","last_clean":"2024-05-09T05:01:41.358343+0000","last_became_active":"2024-05-09T05:01:39.870806+0000","last_became_peered":"2024-05-09T05:01:39.870806+0000","last_unstale":"2024-05-09T05:01:41.358343+0000","last_undegraded":"2024-05-09T05:01:41.358343+0000","last_fullsized":"2024-05-09T05:01:41.358343+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:57:48.604326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358286+0000","last_change":"2024-05-09T05:01:39.865930+0000","last_active":"2024-05-09T05:01:41.358286+0000","last_peered":"2024-05-09T05:01:41.358286+0000","last_clean":"2024-05-09T05:01:41.358286+0000","last_became_active":"2024-05-09T05:01:39.865689+0000","last_became_peered":"2024-05-09T05:01:39.865689+0000","last_unstale":"2024-05-09T05:01:41.358286+0000","last_undegraded":"2024-05-09T05:01:41.358286+0000","last_fullsized":"2024-05-09T05:01:41.358286+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:59:30.203215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.852182+0000","last_change":"2024-05-09T05:01:39.868732+0000","last_active":"2024-05-09T05:01:40.852182+0000","last_peered":"2024-05-09T05:01:40.852182+0000","last_clean":"2024-05-09T05:01:40.852182+0000","last_became_active":"2024-05-09T05:01:39.868433+0000","last_became_peered":"2024-05-09T05:01:39.868433+0000","last_unstale":"2024-05-09T05:01:40.852182+0000","last_undegraded":"2024-05-09T05:01:40.852182+0000","last_fullsized":"2024-05-09T05:01:40.852182+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:20:33.314890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851296+0000","last_change":"2024-05-09T05:01:39.876197+0000","last_active":"2024-05-09T05:01:40.851296+0000","last_peered":"2024-05-09T05:01:40.851296+0000","last_clean":"2024-05-09T05:01:40.851296+0000","last_became_active":"2024-05-09T05:01:39.875857+0000","last_became_peered":"2024-05-09T05:01:39.875857+0000","last_unstale":"2024-05-09T05:01:40.851296+0000","last_undegraded":"2024-05-09T05:01:40.851296+0000","last_fullsized":"2024-05-09T05:01:40.851296+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:42:40.603692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857399+0000","last_change":"2024-05-09T05:01:39.862165+0000","last_active":"2024-05-09T05:01:40.857399+0000","last_peered":"2024-05-09T05:01:40.857399+0000","last_clean":"2024-05-09T05:01:40.857399+0000","last_became_active":"2024-05-09T05:01:39.861827+0000","last_became_peered":"2024-05-09T05:01:39.861827+0000","last_unstale":"2024-05-09T05:01:40.857399+0000","last_undegraded":"2024-05-09T05:01:40.857399+0000","last_fullsized":"2024-05-09T05:01:40.857399+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:06.942055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851402+0000","last_change":"2024-05-09T05:01:39.878415+0000","last_active":"2024-05-09T05:01:40.851402+0000","last_peered":"2024-05-09T05:01:40.851402+0000","last_clean":"2024-05-09T05:01:40.851402+0000","last_became_active":"2024-05-09T05:01:39.878000+0000","last_became_peered":"2024-05-09T05:01:39.878000+0000","last_unstale":"2024-05-09T05:01:40.851402+0000","last_undegraded":"2024-05-09T05:01:40.851402+0000","last_fullsized":"2024-05-09T05:01:40.851402+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:07:34.624914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358065+0000","last_change":"2024-05-09T05:01:39.882587+0000","last_active":"2024-05-09T05:01:41.358065+0000","last_peered":"2024-05-09T05:01:41.358065+0000","last_clean":"2024-05-09T05:01:41.358065+0000","last_became_active":"2024-05-09T05:01:39.880797+0000","last_became_peered":"2024-05-09T05:01:39.880797+0000","last_unstale":"2024-05-09T05:01:41.358065+0000","last_undegraded":"2024-05-09T05:01:41.358065+0000","last_fullsized":"2024-05-09T05:01:41.358065+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:16:55.605416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358313+0000","last_change":"2024-05-09T05:01:39.873638+0000","last_active":"2024-05-09T05:01:41.358313+0000","last_peered":"2024-05-09T05:01:41.358313+0000","last_clean":"2024-05-09T05:01:41.358313+0000","last_became_active":"2024-05-09T05:01:39.873367+0000","last_became_peered":"2024-05-09T05:01:39.873367+0000","last_unstale":"2024-05-09T05:01:41.358313+0000","last_undegraded":"2024-05-09T05:01:41.358313+0000","last_fullsized":"2024-05-09T05:01:41.358313+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:21.336536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"50'4","reported_seq":18,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358595+0000","last_change":"2024-05-09T05:01:38.861502+0000","last_active":"2024-05-09T05:01:41.358595+0000","last_peered":"2024-05-09T05:01:41.358595+0000","last_clean":"2024-05-09T05:01:41.358595+0000","last_became_active":"2024-05-09T05:01:38.861131+0000","last_became_peered":"2024-05-09T05:01:38.861131+0000","last_unstale":"2024-05-09T05:01:41.358595+0000","last_undegraded":"2024-05-09T05:01:41.358595+0000","last_fullsized":"2024-05-09T05:01:41.358595+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:47:15.947904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358929+0000","last_change":"2024-05-09T05:01:39.878999+0000","last_active":"2024-05-09T05:01:41.358929+0000","last_peered":"2024-05-09T05:01:41.358929+0000","last_clean":"2024-05-09T05:01:41.358929+0000","last_became_active":"2024-05-09T05:01:39.874591+0000","last_became_peered":"2024-05-09T05:01:39.874591+0000","last_unstale":"2024-05-09T05:01:41.358929+0000","last_undegraded":"2024-05-09T05:01:41.358929+0000","last_fullsized":"2024-05-09T05:01:41.358929+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:07:16.662667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851161+0000","last_change":"2024-05-09T05:01:38.862945+0000","last_active":"2024-05-09T05:01:40.851161+0000","last_peered":"2024-05-09T05:01:40.851161+0000","last_clean":"2024-05-09T05:01:40.851161+0000","last_became_active":"2024-05-09T05:01:38.862633+0000","last_became_peered":"2024-05-09T05:01:38.862633+0000","last_unstale":"2024-05-09T05:01:40.851161+0000","last_undegraded":"2024-05-09T05:01:40.851161+0000","last_fullsized":"2024-05-09T05:01:40.851161+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:39:03.448180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358625+0000","last_change":"2024-05-09T05:01:39.862888+0000","last_active":"2024-05-09T05:01:41.358625+0000","last_peered":"2024-05-09T05:01:41.358625+0000","last_clean":"2024-05-09T05:01:41.358625+0000","last_became_active":"2024-05-09T05:01:39.862585+0000","last_became_peered":"2024-05-09T05:01:39.862585+0000","last_unstale":"2024-05-09T05:01:41.358625+0000","last_undegraded":"2024-05-09T05:01:41.358625+0000","last_fullsized":"2024-05-09T05:01:41.358625+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:26.273557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358718+0000","last_change":"2024-05-09T05:01:38.857444+0000","last_active":"2024-05-09T05:01:41.358718+0000","last_peered":"2024-05-09T05:01:41.358718+0000","last_clean":"2024-05-09T05:01:41.358718+0000","last_became_active":"2024-05-09T05:01:38.857133+0000","last_became_peered":"2024-05-09T05:01:38.857133+0000","last_unstale":"2024-05-09T05:01:41.358718+0000","last_undegraded":"2024-05-09T05:01:41.358718+0000","last_fullsized":"2024-05-09T05:01:41.358718+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:39:57.155429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358464+0000","last_change":"2024-05-09T05:01:39.870882+0000","last_active":"2024-05-09T05:01:41.358464+0000","last_peered":"2024-05-09T05:01:41.358464+0000","last_clean":"2024-05-09T05:01:41.358464+0000","last_became_active":"2024-05-09T05:01:39.870567+0000","last_became_peered":"2024-05-09T05:01:39.870567+0000","last_unstale":"2024-05-09T05:01:41.358464+0000","last_undegraded":"2024-05-09T05:01:41.358464+0000","last_fullsized":"2024-05-09T05:01:41.358464+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:21.186391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358528+0000","last_change":"2024-05-09T05:01:38.863364+0000","last_active":"2024-05-09T05:01:41.358528+0000","last_peered":"2024-05-09T05:01:41.358528+0000","last_clean":"2024-05-09T05:01:41.358528+0000","last_became_active":"2024-05-09T05:01:38.863031+0000","last_became_peered":"2024-05-09T05:01:38.863031+0000","last_unstale":"2024-05-09T05:01:41.358528+0000","last_undegraded":"2024-05-09T05:01:41.358528+0000","last_fullsized":"2024-05-09T05:01:41.358528+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:53:39.994209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.850817+0000","last_change":"2024-05-09T05:01:39.873520+0000","last_active":"2024-05-09T05:01:40.850817+0000","last_peered":"2024-05-09T05:01:40.850817+0000","last_clean":"2024-05-09T05:01:40.850817+0000","last_became_active":"2024-05-09T05:01:39.872905+0000","last_became_peered":"2024-05-09T05:01:39.872905+0000","last_unstale":"2024-05-09T05:01:40.850817+0000","last_undegraded":"2024-05-09T05:01:40.850817+0000","last_fullsized":"2024-05-09T05:01:40.850817+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:37:47.075587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980479+0000","last_change":"2024-05-09T05:01:38.862527+0000","last_active":"2024-05-09T05:01:40.980479+0000","last_peered":"2024-05-09T05:01:40.980479+0000","last_clean":"2024-05-09T05:01:40.980479+0000","last_became_active":"2024-05-09T05:01:38.861979+0000","last_became_peered":"2024-05-09T05:01:38.861979+0000","last_unstale":"2024-05-09T05:01:40.980479+0000","last_undegraded":"2024-05-09T05:01:40.980479+0000","last_fullsized":"2024-05-09T05:01:40.980479+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:31:57.936029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851068+0000","last_change":"2024-05-09T05:01:39.865218+0000","last_active":"2024-05-09T05:01:40.851068+0000","last_peered":"2024-05-09T05:01:40.851068+0000","last_clean":"2024-05-09T05:01:40.851068+0000","last_became_active":"2024-05-09T05:01:39.864965+0000","last_became_peered":"2024-05-09T05:01:39.864965+0000","last_unstale":"2024-05-09T05:01:40.851068+0000","last_undegraded":"2024-05-09T05:01:40.851068+0000","last_fullsized":"2024-05-09T05:01:40.851068+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:50:38.445530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358239+0000","last_change":"2024-05-09T05:01:38.865608+0000","last_active":"2024-05-09T05:01:41.358239+0000","last_peered":"2024-05-09T05:01:41.358239+0000","last_clean":"2024-05-09T05:01:41.358239+0000","last_became_active":"2024-05-09T05:01:38.865155+0000","last_became_peered":"2024-05-09T05:01:38.865155+0000","last_unstale":"2024-05-09T05:01:41.358239+0000","last_undegraded":"2024-05-09T05:01:41.358239+0000","last_fullsized":"2024-05-09T05:01:41.358239+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:22:23.371054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.357800+0000","last_change":"2024-05-09T05:01:39.862637+0000","last_active":"2024-05-09T05:01:41.357800+0000","last_peered":"2024-05-09T05:01:41.357800+0000","last_clean":"2024-05-09T05:01:41.357800+0000","last_became_active":"2024-05-09T05:01:39.862059+0000","last_became_peered":"2024-05-09T05:01:39.862059+0000","last_unstale":"2024-05-09T05:01:41.357800+0000","last_undegraded":"2024-05-09T05:01:41.357800+0000","last_fullsized":"2024-05-09T05:01:41.357800+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:51:43.777982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358236+0000","last_change":"2024-05-09T05:01:38.865473+0000","last_active":"2024-05-09T05:01:41.358236+0000","last_peered":"2024-05-09T05:01:41.358236+0000","last_clean":"2024-05-09T05:01:41.358236+0000","last_became_active":"2024-05-09T05:01:38.864641+0000","last_became_peered":"2024-05-09T05:01:38.864641+0000","last_unstale":"2024-05-09T05:01:41.358236+0000","last_undegraded":"2024-05-09T05:01:41.358236+0000","last_fullsized":"2024-05-09T05:01:41.358236+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:39.777738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.850858+0000","last_change":"2024-05-09T05:01:39.871696+0000","last_active":"2024-05-09T05:01:40.850858+0000","last_peered":"2024-05-09T05:01:40.850858+0000","last_clean":"2024-05-09T05:01:40.850858+0000","last_became_active":"2024-05-09T05:01:39.871423+0000","last_became_peered":"2024-05-09T05:01:39.871423+0000","last_unstale":"2024-05-09T05:01:40.850858+0000","last_undegraded":"2024-05-09T05:01:40.850858+0000","last_fullsized":"2024-05-09T05:01:40.850858+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:03:16.895228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980459+0000","last_change":"2024-05-09T05:01:38.866604+0000","last_active":"2024-05-09T05:01:40.980459+0000","last_peered":"2024-05-09T05:01:40.980459+0000","last_clean":"2024-05-09T05:01:40.980459+0000","last_became_active":"2024-05-09T05:01:38.866324+0000","last_became_peered":"2024-05-09T05:01:38.866324+0000","last_unstale":"2024-05-09T05:01:40.980459+0000","last_undegraded":"2024-05-09T05:01:40.980459+0000","last_fullsized":"2024-05-09T05:01:40.980459+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:22.170335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980784+0000","last_change":"2024-05-09T05:01:39.866146+0000","last_active":"2024-05-09T05:01:40.980784+0000","last_peered":"2024-05-09T05:01:40.980784+0000","last_clean":"2024-05-09T05:01:40.980784+0000","last_became_active":"2024-05-09T05:01:39.865751+0000","last_became_peered":"2024-05-09T05:01:39.865751+0000","last_unstale":"2024-05-09T05:01:40.980784+0000","last_undegraded":"2024-05-09T05:01:40.980784+0000","last_fullsized":"2024-05-09T05:01:40.980784+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:30.919586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857772+0000","last_change":"2024-05-09T05:01:38.858711+0000","last_active":"2024-05-09T05:01:40.857772+0000","last_peered":"2024-05-09T05:01:40.857772+0000","last_clean":"2024-05-09T05:01:40.857772+0000","last_became_active":"2024-05-09T05:01:38.858438+0000","last_became_peered":"2024-05-09T05:01:38.858438+0000","last_unstale":"2024-05-09T05:01:40.857772+0000","last_undegraded":"2024-05-09T05:01:40.857772+0000","last_fullsized":"2024-05-09T05:01:40.857772+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:45.223268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857777+0000","last_change":"2024-05-09T05:01:39.881199+0000","last_active":"2024-05-09T05:01:40.857777+0000","last_peered":"2024-05-09T05:01:40.857777+0000","last_clean":"2024-05-09T05:01:40.857777+0000","last_became_active":"2024-05-09T05:01:39.880897+0000","last_became_peered":"2024-05-09T05:01:39.880897+0000","last_unstale":"2024-05-09T05:01:40.857777+0000","last_undegraded":"2024-05-09T05:01:40.857777+0000","last_fullsized":"2024-05-09T05:01:40.857777+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:03:16.895228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851249+0000","last_change":"2024-05-09T05:01:38.860738+0000","last_active":"2024-05-09T05:01:40.851249+0000","last_peered":"2024-05-09T05:01:40.851249+0000","last_clean":"2024-05-09T05:01:40.851249+0000","last_became_active":"2024-05-09T05:01:38.860472+0000","last_became_peered":"2024-05-09T05:01:38.860472+0000","last_unstale":"2024-05-09T05:01:40.851249+0000","last_undegraded":"2024-05-09T05:01:40.851249+0000","last_fullsized":"2024-05-09T05:01:40.851249+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:03.504627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358890+0000","last_change":"2024-05-09T05:01:39.874632+0000","last_active":"2024-05-09T05:01:41.358890+0000","last_peered":"2024-05-09T05:01:41.358890+0000","last_clean":"2024-05-09T05:01:41.358890+0000","last_became_active":"2024-05-09T05:01:39.871669+0000","last_became_peered":"2024-05-09T05:01:39.871669+0000","last_unstale":"2024-05-09T05:01:41.358890+0000","last_undegraded":"2024-05-09T05:01:41.358890+0000","last_fullsized":"2024-05-09T05:01:41.358890+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:22:24.375967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"50'3","reported_seq":17,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857601+0000","last_change":"2024-05-09T05:01:38.864487+0000","last_active":"2024-05-09T05:01:40.857601+0000","last_peered":"2024-05-09T05:01:40.857601+0000","last_clean":"2024-05-09T05:01:40.857601+0000","last_became_active":"2024-05-09T05:01:38.864194+0000","last_became_peered":"2024-05-09T05:01:38.864194+0000","last_unstale":"2024-05-09T05:01:40.857601+0000","last_undegraded":"2024-05-09T05:01:40.857601+0000","last_fullsized":"2024-05-09T05:01:40.857601+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:26:10.131662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857473+0000","last_change":"2024-05-09T05:01:39.862711+0000","last_active":"2024-05-09T05:01:40.857473+0000","last_peered":"2024-05-09T05:01:40.857473+0000","last_clean":"2024-05-09T05:01:40.857473+0000","last_became_active":"2024-05-09T05:01:39.862322+0000","last_became_peered":"2024-05-09T05:01:39.862322+0000","last_unstale":"2024-05-09T05:01:40.857473+0000","last_undegraded":"2024-05-09T05:01:40.857473+0000","last_fullsized":"2024-05-09T05:01:40.857473+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:48:03.359391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"50'6","reported_seq":20,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980752+0000","last_change":"2024-05-09T05:01:38.868251+0000","last_active":"2024-05-09T05:01:40.980752+0000","last_peered":"2024-05-09T05:01:40.980752+0000","last_clean":"2024-05-09T05:01:40.980752+0000","last_became_active":"2024-05-09T05:01:38.868026+0000","last_became_peered":"2024-05-09T05:01:38.868026+0000","last_unstale":"2024-05-09T05:01:40.980752+0000","last_undegraded":"2024-05-09T05:01:40.980752+0000","last_fullsized":"2024-05-09T05:01:40.980752+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:30:01.506489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.357854+0000","last_change":"2024-05-09T05:01:39.869503+0000","last_active":"2024-05-09T05:01:41.357854+0000","last_peered":"2024-05-09T05:01:41.357854+0000","last_clean":"2024-05-09T05:01:41.357854+0000","last_became_active":"2024-05-09T05:01:39.868873+0000","last_became_peered":"2024-05-09T05:01:39.868873+0000","last_unstale":"2024-05-09T05:01:41.357854+0000","last_undegraded":"2024-05-09T05:01:41.357854+0000","last_fullsized":"2024-05-09T05:01:41.357854+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:21.336536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851168+0000","last_change":"2024-05-09T05:01:38.859004+0000","last_active":"2024-05-09T05:01:40.851168+0000","last_peered":"2024-05-09T05:01:40.851168+0000","last_clean":"2024-05-09T05:01:40.851168+0000","last_became_active":"2024-05-09T05:01:38.858712+0000","last_became_peered":"2024-05-09T05:01:38.858712+0000","last_unstale":"2024-05-09T05:01:40.851168+0000","last_undegraded":"2024-05-09T05:01:40.851168+0000","last_fullsized":"2024-05-09T05:01:40.851168+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:58:53.883816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857379+0000","last_change":"2024-05-09T05:01:39.863597+0000","last_active":"2024-05-09T05:01:40.857379+0000","last_peered":"2024-05-09T05:01:40.857379+0000","last_clean":"2024-05-09T05:01:40.857379+0000","last_became_active":"2024-05-09T05:01:39.863310+0000","last_became_peered":"2024-05-09T05:01:39.863310+0000","last_unstale":"2024-05-09T05:01:40.857379+0000","last_undegraded":"2024-05-09T05:01:40.857379+0000","last_fullsized":"2024-05-09T05:01:40.857379+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:21.186391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.357824+0000","last_change":"2024-05-09T05:01:38.860319+0000","last_active":"2024-05-09T05:01:41.357824+0000","last_peered":"2024-05-09T05:01:41.357824+0000","last_clean":"2024-05-09T05:01:41.357824+0000","last_became_active":"2024-05-09T05:01:38.860033+0000","last_became_peered":"2024-05-09T05:01:38.860033+0000","last_unstale":"2024-05-09T05:01:41.357824+0000","last_undegraded":"2024-05-09T05:01:41.357824+0000","last_fullsized":"2024-05-09T05:01:41.357824+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:05.615229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980511+0000","last_change":"2024-05-09T05:01:39.860133+0000","last_active":"2024-05-09T05:01:40.980511+0000","last_peered":"2024-05-09T05:01:40.980511+0000","last_clean":"2024-05-09T05:01:40.980511+0000","last_became_active":"2024-05-09T05:01:39.859806+0000","last_became_peered":"2024-05-09T05:01:39.859806+0000","last_unstale":"2024-05-09T05:01:40.980511+0000","last_undegraded":"2024-05-09T05:01:40.980511+0000","last_fullsized":"2024-05-09T05:01:40.980511+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:35.258317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358372+0000","last_change":"2024-05-09T05:01:38.860319+0000","last_active":"2024-05-09T05:01:41.358372+0000","last_peered":"2024-05-09T05:01:41.358372+0000","last_clean":"2024-05-09T05:01:41.358372+0000","last_became_active":"2024-05-09T05:01:38.860025+0000","last_became_peered":"2024-05-09T05:01:38.860025+0000","last_unstale":"2024-05-09T05:01:41.358372+0000","last_undegraded":"2024-05-09T05:01:41.358372+0000","last_fullsized":"2024-05-09T05:01:41.358372+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:35:19.523791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857419+0000","last_change":"2024-05-09T05:01:39.872694+0000","last_active":"2024-05-09T05:01:40.857419+0000","last_peered":"2024-05-09T05:01:40.857419+0000","last_clean":"2024-05-09T05:01:40.857419+0000","last_became_active":"2024-05-09T05:01:39.872386+0000","last_became_peered":"2024-05-09T05:01:39.872386+0000","last_unstale":"2024-05-09T05:01:40.857419+0000","last_undegraded":"2024-05-09T05:01:40.857419+0000","last_fullsized":"2024-05-09T05:01:40.857419+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:26.273557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"50'8","reported_seq":22,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857335+0000","last_change":"2024-05-09T05:01:38.863353+0000","last_active":"2024-05-09T05:01:40.857335+0000","last_peered":"2024-05-09T05:01:40.857335+0000","last_clean":"2024-05-09T05:01:40.857335+0000","last_became_active":"2024-05-09T05:01:38.863084+0000","last_became_peered":"2024-05-09T05:01:38.863084+0000","last_unstale":"2024-05-09T05:01:40.857335+0000","last_undegraded":"2024-05-09T05:01:40.857335+0000","last_fullsized":"2024-05-09T05:01:40.857335+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:39:57.155429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857692+0000","last_change":"2024-05-09T05:01:39.868252+0000","last_active":"2024-05-09T05:01:40.857692+0000","last_peered":"2024-05-09T05:01:40.857692+0000","last_clean":"2024-05-09T05:01:40.857692+0000","last_became_active":"2024-05-09T05:01:39.867852+0000","last_became_peered":"2024-05-09T05:01:39.867852+0000","last_unstale":"2024-05-09T05:01:40.857692+0000","last_undegraded":"2024-05-09T05:01:40.857692+0000","last_fullsized":"2024-05-09T05:01:40.857692+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:37:47.075587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851773+0000","last_change":"2024-05-09T05:01:38.858925+0000","last_active":"2024-05-09T05:01:40.851773+0000","last_peered":"2024-05-09T05:01:40.851773+0000","last_clean":"2024-05-09T05:01:40.851773+0000","last_became_active":"2024-05-09T05:01:38.858625+0000","last_became_peered":"2024-05-09T05:01:38.858625+0000","last_unstale":"2024-05-09T05:01:40.851773+0000","last_undegraded":"2024-05-09T05:01:40.851773+0000","last_fullsized":"2024-05-09T05:01:40.851773+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:17:28.288161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358268+0000","last_change":"2024-05-09T05:01:39.880829+0000","last_active":"2024-05-09T05:01:41.358268+0000","last_peered":"2024-05-09T05:01:41.358268+0000","last_clean":"2024-05-09T05:01:41.358268+0000","last_became_active":"2024-05-09T05:01:39.880452+0000","last_became_peered":"2024-05-09T05:01:39.880452+0000","last_unstale":"2024-05-09T05:01:41.358268+0000","last_undegraded":"2024-05-09T05:01:41.358268+0000","last_fullsized":"2024-05-09T05:01:41.358268+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:21.186391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358473+0000","last_change":"2024-05-09T05:01:38.860324+0000","last_active":"2024-05-09T05:01:41.358473+0000","last_peered":"2024-05-09T05:01:41.358473+0000","last_clean":"2024-05-09T05:01:41.358473+0000","last_became_active":"2024-05-09T05:01:38.860006+0000","last_became_peered":"2024-05-09T05:01:38.860006+0000","last_unstale":"2024-05-09T05:01:41.358473+0000","last_undegraded":"2024-05-09T05:01:41.358473+0000","last_fullsized":"2024-05-09T05:01:41.358473+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:07:15.657753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851342+0000","last_change":"2024-05-09T05:01:39.860722+0000","last_active":"2024-05-09T05:01:40.851342+0000","last_peered":"2024-05-09T05:01:40.851342+0000","last_clean":"2024-05-09T05:01:40.851342+0000","last_became_active":"2024-05-09T05:01:39.858607+0000","last_became_peered":"2024-05-09T05:01:39.858607+0000","last_unstale":"2024-05-09T05:01:40.851342+0000","last_undegraded":"2024-05-09T05:01:40.851342+0000","last_fullsized":"2024-05-09T05:01:40.851342+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:48:49.629108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358453+0000","last_change":"2024-05-09T05:01:38.860226+0000","last_active":"2024-05-09T05:01:41.358453+0000","last_peered":"2024-05-09T05:01:41.358453+0000","last_clean":"2024-05-09T05:01:41.358453+0000","last_became_active":"2024-05-09T05:01:38.859683+0000","last_became_peered":"2024-05-09T05:01:38.859683+0000","last_unstale":"2024-05-09T05:01:41.358453+0000","last_undegraded":"2024-05-09T05:01:41.358453+0000","last_fullsized":"2024-05-09T05:01:41.358453+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:00:56.428799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857659+0000","last_change":"2024-05-09T05:01:39.862901+0000","last_active":"2024-05-09T05:01:40.857659+0000","last_peered":"2024-05-09T05:01:40.857659+0000","last_clean":"2024-05-09T05:01:40.857659+0000","last_became_active":"2024-05-09T05:01:39.862621+0000","last_became_peered":"2024-05-09T05:01:39.862621+0000","last_unstale":"2024-05-09T05:01:40.857659+0000","last_undegraded":"2024-05-09T05:01:40.857659+0000","last_fullsized":"2024-05-09T05:01:40.857659+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:17:38.783370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358545+0000","last_change":"2024-05-09T05:01:38.861565+0000","last_active":"2024-05-09T05:01:41.358545+0000","last_peered":"2024-05-09T05:01:41.358545+0000","last_clean":"2024-05-09T05:01:41.358545+0000","last_became_active":"2024-05-09T05:01:38.861245+0000","last_became_peered":"2024-05-09T05:01:38.861245+0000","last_unstale":"2024-05-09T05:01:41.358545+0000","last_undegraded":"2024-05-09T05:01:41.358545+0000","last_fullsized":"2024-05-09T05:01:41.358545+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:45.223268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358339+0000","last_change":"2024-05-09T05:01:39.869735+0000","last_active":"2024-05-09T05:01:41.358339+0000","last_peered":"2024-05-09T05:01:41.358339+0000","last_clean":"2024-05-09T05:01:41.358339+0000","last_became_active":"2024-05-09T05:01:39.869239+0000","last_became_peered":"2024-05-09T05:01:39.869239+0000","last_unstale":"2024-05-09T05:01:41.358339+0000","last_undegraded":"2024-05-09T05:01:41.358339+0000","last_fullsized":"2024-05-09T05:01:41.358339+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:26.273557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851280+0000","last_change":"2024-05-09T05:01:38.868383+0000","last_active":"2024-05-09T05:01:40.851280+0000","last_peered":"2024-05-09T05:01:40.851280+0000","last_clean":"2024-05-09T05:01:40.851280+0000","last_became_active":"2024-05-09T05:01:38.867935+0000","last_became_peered":"2024-05-09T05:01:38.867935+0000","last_unstale":"2024-05-09T05:01:40.851280+0000","last_undegraded":"2024-05-09T05:01:40.851280+0000","last_fullsized":"2024-05-09T05:01:40.851280+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:09:43.953892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358624+0000","last_change":"2024-05-09T05:01:39.868847+0000","last_active":"2024-05-09T05:01:41.358624+0000","last_peered":"2024-05-09T05:01:41.358624+0000","last_clean":"2024-05-09T05:01:41.358624+0000","last_became_active":"2024-05-09T05:01:39.868602+0000","last_became_peered":"2024-05-09T05:01:39.868602+0000","last_unstale":"2024-05-09T05:01:41.358624+0000","last_undegraded":"2024-05-09T05:01:41.358624+0000","last_fullsized":"2024-05-09T05:01:41.358624+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:57:48.604326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358419+0000","last_change":"2024-05-09T05:01:38.861303+0000","last_active":"2024-05-09T05:01:41.358419+0000","last_peered":"2024-05-09T05:01:41.358419+0000","last_clean":"2024-05-09T05:01:41.358419+0000","last_became_active":"2024-05-09T05:01:38.860616+0000","last_became_peered":"2024-05-09T05:01:38.860616+0000","last_unstale":"2024-05-09T05:01:41.358419+0000","last_undegraded":"2024-05-09T05:01:41.358419+0000","last_fullsized":"2024-05-09T05:01:41.358419+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:52.445863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358527+0000","last_change":"2024-05-09T05:01:39.875323+0000","last_active":"2024-05-09T05:01:41.358527+0000","last_peered":"2024-05-09T05:01:41.358527+0000","last_clean":"2024-05-09T05:01:41.358527+0000","last_became_active":"2024-05-09T05:01:39.875028+0000","last_became_peered":"2024-05-09T05:01:39.875028+0000","last_unstale":"2024-05-09T05:01:41.358527+0000","last_undegraded":"2024-05-09T05:01:41.358527+0000","last_fullsized":"2024-05-09T05:01:41.358527+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:06.942055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.357925+0000","last_change":"2024-05-09T05:01:38.866624+0000","last_active":"2024-05-09T05:01:41.357925+0000","last_peered":"2024-05-09T05:01:41.357925+0000","last_clean":"2024-05-09T05:01:41.357925+0000","last_became_active":"2024-05-09T05:01:38.866301+0000","last_became_peered":"2024-05-09T05:01:38.866301+0000","last_unstale":"2024-05-09T05:01:41.357925+0000","last_undegraded":"2024-05-09T05:01:41.357925+0000","last_fullsized":"2024-05-09T05:01:41.357925+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:26:10.131662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358294+0000","last_change":"2024-05-09T05:01:39.874994+0000","last_active":"2024-05-09T05:01:41.358294+0000","last_peered":"2024-05-09T05:01:41.358294+0000","last_clean":"2024-05-09T05:01:41.358294+0000","last_became_active":"2024-05-09T05:01:39.874713+0000","last_became_peered":"2024-05-09T05:01:39.874713+0000","last_unstale":"2024-05-09T05:01:41.358294+0000","last_undegraded":"2024-05-09T05:01:41.358294+0000","last_fullsized":"2024-05-09T05:01:41.358294+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:06.942055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358969+0000","last_change":"2024-05-09T05:01:38.867281+0000","last_active":"2024-05-09T05:01:41.358969+0000","last_peered":"2024-05-09T05:01:41.358969+0000","last_clean":"2024-05-09T05:01:41.358969+0000","last_became_active":"2024-05-09T05:01:38.867007+0000","last_became_peered":"2024-05-09T05:01:38.867007+0000","last_unstale":"2024-05-09T05:01:41.358969+0000","last_undegraded":"2024-05-09T05:01:41.358969+0000","last_fullsized":"2024-05-09T05:01:41.358969+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:28.819242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358407+0000","last_change":"2024-05-09T05:01:39.873250+0000","last_active":"2024-05-09T05:01:41.358407+0000","last_peered":"2024-05-09T05:01:41.358407+0000","last_clean":"2024-05-09T05:01:41.358407+0000","last_became_active":"2024-05-09T05:01:39.872934+0000","last_became_peered":"2024-05-09T05:01:39.872934+0000","last_unstale":"2024-05-09T05:01:41.358407+0000","last_undegraded":"2024-05-09T05:01:41.358407+0000","last_fullsized":"2024-05-09T05:01:41.358407+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:18:54.996982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"51'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.856206+0000","last_change":"2024-05-09T05:01:38.862288+0000","last_active":"2024-05-09T05:01:41.856206+0000","last_peered":"2024-05-09T05:01:41.856206+0000","last_clean":"2024-05-09T05:01:41.856206+0000","last_became_active":"2024-05-09T05:01:38.862002+0000","last_became_peered":"2024-05-09T05:01:38.862002+0000","last_unstale":"2024-05-09T05:01:41.856206+0000","last_undegraded":"2024-05-09T05:01:41.856206+0000","last_fullsized":"2024-05-09T05:01:41.856206+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:05:21.818769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358948+0000","last_change":"2024-05-09T05:01:39.878889+0000","last_active":"2024-05-09T05:01:41.358948+0000","last_peered":"2024-05-09T05:01:41.358948+0000","last_clean":"2024-05-09T05:01:41.358948+0000","last_became_active":"2024-05-09T05:01:39.874324+0000","last_became_peered":"2024-05-09T05:01:39.874324+0000","last_unstale":"2024-05-09T05:01:41.358948+0000","last_undegraded":"2024-05-09T05:01:41.358948+0000","last_fullsized":"2024-05-09T05:01:41.358948+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:39:58.160343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358466+0000","last_change":"2024-05-09T05:01:38.858855+0000","last_active":"2024-05-09T05:01:41.358466+0000","last_peered":"2024-05-09T05:01:41.358466+0000","last_clean":"2024-05-09T05:01:41.358466+0000","last_became_active":"2024-05-09T05:01:38.858570+0000","last_became_peered":"2024-05-09T05:01:38.858570+0000","last_unstale":"2024-05-09T05:01:41.358466+0000","last_undegraded":"2024-05-09T05:01:41.358466+0000","last_fullsized":"2024-05-09T05:01:41.358466+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:38.746187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358661+0000","last_change":"2024-05-09T05:01:39.870147+0000","last_active":"2024-05-09T05:01:41.358661+0000","last_peered":"2024-05-09T05:01:41.358661+0000","last_clean":"2024-05-09T05:01:41.358661+0000","last_became_active":"2024-05-09T05:01:39.869848+0000","last_became_peered":"2024-05-09T05:01:39.869848+0000","last_unstale":"2024-05-09T05:01:41.358661+0000","last_undegraded":"2024-05-09T05:01:41.358661+0000","last_fullsized":"2024-05-09T05:01:41.358661+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:48:03.359391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358583+0000","last_change":"2024-05-09T05:01:38.865398+0000","last_active":"2024-05-09T05:01:41.358583+0000","last_peered":"2024-05-09T05:01:41.358583+0000","last_clean":"2024-05-09T05:01:41.358583+0000","last_became_active":"2024-05-09T05:01:38.865098+0000","last_became_peered":"2024-05-09T05:01:38.865098+0000","last_unstale":"2024-05-09T05:01:41.358583+0000","last_undegraded":"2024-05-09T05:01:41.358583+0000","last_fullsized":"2024-05-09T05:01:41.358583+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:17:00.452604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851102+0000","last_change":"2024-05-09T05:01:39.863602+0000","last_active":"2024-05-09T05:01:40.851102+0000","last_peered":"2024-05-09T05:01:40.851102+0000","last_clean":"2024-05-09T05:01:40.851102+0000","last_became_active":"2024-05-09T05:01:39.863245+0000","last_became_peered":"2024-05-09T05:01:39.863245+0000","last_unstale":"2024-05-09T05:01:40.851102+0000","last_undegraded":"2024-05-09T05:01:40.851102+0000","last_fullsized":"2024-05-09T05:01:40.851102+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:33:27.012550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358486+0000","last_change":"2024-05-09T05:01:38.863121+0000","last_active":"2024-05-09T05:01:41.358486+0000","last_peered":"2024-05-09T05:01:41.358486+0000","last_clean":"2024-05-09T05:01:41.358486+0000","last_became_active":"2024-05-09T05:01:38.862843+0000","last_became_peered":"2024-05-09T05:01:38.862843+0000","last_unstale":"2024-05-09T05:01:41.358486+0000","last_undegraded":"2024-05-09T05:01:41.358486+0000","last_fullsized":"2024-05-09T05:01:41.358486+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:05.615229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857557+0000","last_change":"2024-05-09T05:01:39.863381+0000","last_active":"2024-05-09T05:01:40.857557+0000","last_peered":"2024-05-09T05:01:40.857557+0000","last_clean":"2024-05-09T05:01:40.857557+0000","last_became_active":"2024-05-09T05:01:39.862839+0000","last_became_peered":"2024-05-09T05:01:39.862839+0000","last_unstale":"2024-05-09T05:01:40.857557+0000","last_undegraded":"2024-05-09T05:01:40.857557+0000","last_fullsized":"2024-05-09T05:01:40.857557+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:18:54.996982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980497+0000","last_change":"2024-05-09T05:01:38.866869+0000","last_active":"2024-05-09T05:01:40.980497+0000","last_peered":"2024-05-09T05:01:40.980497+0000","last_clean":"2024-05-09T05:01:40.980497+0000","last_became_active":"2024-05-09T05:01:38.866569+0000","last_became_peered":"2024-05-09T05:01:38.866569+0000","last_unstale":"2024-05-09T05:01:40.980497+0000","last_undegraded":"2024-05-09T05:01:40.980497+0000","last_fullsized":"2024-05-09T05:01:40.980497+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:14:47.456838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.357865+0000","last_change":"2024-05-09T05:01:39.879631+0000","last_active":"2024-05-09T05:01:41.357865+0000","last_peered":"2024-05-09T05:01:41.357865+0000","last_clean":"2024-05-09T05:01:41.357865+0000","last_became_active":"2024-05-09T05:01:39.875087+0000","last_became_peered":"2024-05-09T05:01:39.875087+0000","last_unstale":"2024-05-09T05:01:41.357865+0000","last_undegraded":"2024-05-09T05:01:41.357865+0000","last_fullsized":"2024-05-09T05:01:41.357865+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:30.919586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.357900+0000","last_change":"2024-05-09T05:01:38.859275+0000","last_active":"2024-05-09T05:01:41.357900+0000","last_peered":"2024-05-09T05:01:41.357900+0000","last_clean":"2024-05-09T05:01:41.357900+0000","last_became_active":"2024-05-09T05:01:38.858976+0000","last_became_peered":"2024-05-09T05:01:38.858976+0000","last_unstale":"2024-05-09T05:01:41.357900+0000","last_undegraded":"2024-05-09T05:01:41.357900+0000","last_fullsized":"2024-05-09T05:01:41.357900+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:48:03.567166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.850893+0000","last_change":"2024-05-09T05:01:39.870972+0000","last_active":"2024-05-09T05:01:40.850893+0000","last_peered":"2024-05-09T05:01:40.850893+0000","last_clean":"2024-05-09T05:01:40.850893+0000","last_became_active":"2024-05-09T05:01:39.867594+0000","last_became_peered":"2024-05-09T05:01:39.867594+0000","last_unstale":"2024-05-09T05:01:40.850893+0000","last_undegraded":"2024-05-09T05:01:40.850893+0000","last_fullsized":"2024-05-09T05:01:40.850893+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:23:38.645452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851033+0000","last_change":"2024-05-09T05:01:38.865444+0000","last_active":"2024-05-09T05:01:40.851033+0000","last_peered":"2024-05-09T05:01:40.851033+0000","last_clean":"2024-05-09T05:01:40.851033+0000","last_became_active":"2024-05-09T05:01:38.865025+0000","last_became_peered":"2024-05-09T05:01:38.865025+0000","last_unstale":"2024-05-09T05:01:40.851033+0000","last_undegraded":"2024-05-09T05:01:40.851033+0000","last_fullsized":"2024-05-09T05:01:40.851033+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:39.777738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851043+0000","last_change":"2024-05-09T05:01:39.867490+0000","last_active":"2024-05-09T05:01:40.851043+0000","last_peered":"2024-05-09T05:01:40.851043+0000","last_clean":"2024-05-09T05:01:40.851043+0000","last_became_active":"2024-05-09T05:01:39.867145+0000","last_became_peered":"2024-05-09T05:01:39.867145+0000","last_unstale":"2024-05-09T05:01:40.851043+0000","last_undegraded":"2024-05-09T05:01:40.851043+0000","last_fullsized":"2024-05-09T05:01:40.851043+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:08:54.716934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.357792+0000","last_change":"2024-05-09T05:01:38.863680+0000","last_active":"2024-05-09T05:01:41.357792+0000","last_peered":"2024-05-09T05:01:41.357792+0000","last_clean":"2024-05-09T05:01:41.357792+0000","last_became_active":"2024-05-09T05:01:38.863378+0000","last_became_peered":"2024-05-09T05:01:38.863378+0000","last_unstale":"2024-05-09T05:01:41.357792+0000","last_undegraded":"2024-05-09T05:01:41.357792+0000","last_fullsized":"2024-05-09T05:01:41.357792+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:06:28.931921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":167,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857137+0000","last_change":"2024-05-09T05:00:08.667158+0000","last_active":"2024-05-09T05:01:40.857137+0000","last_peered":"2024-05-09T05:01:40.857137+0000","last_clean":"2024-05-09T05:01:40.857137+0000","last_became_active":"2024-05-09T05:00:08.666316+0000","last_became_peered":"2024-05-09T05:00:08.666316+0000","last_unstale":"2024-05-09T05:01:40.857137+0000","last_undegraded":"2024-05-09T05:01:40.857137+0000","last_fullsized":"2024-05-09T05:01:40.857137+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_clean_scrub_stamp":"2024-05-09T05:00:07.370119+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:20.234643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28204,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343636,"statfs":{"total":96636764160,"available":96607883264,"internally_reserved":0,"allocated":1212416,"data_stored":861719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":598016,"data_stored":264499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":265151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":622592,"data_stored":272803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31784,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340056,"statfs":{"total":96636764160,"available":96604217344,"internally_reserved":0,"allocated":618496,"data_stored":271247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":618496,"data_stored":271247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31780,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340060,"statfs":{"total":96636764160,"available":96604221440,"internally_reserved":0,"allocated":614400,"data_stored":271281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1220608,"data_stored":863753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:01:43.915 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:01:44.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.055+0000 7fcaaa3e7640 1 Processor -- start 2024-05-09T05:01:44.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.055+0000 7fcaaa3e7640 1 -- start start 2024-05-09T05:01:44.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.056+0000 7fcaaa3e7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcaa415f3c0 0x7fcaa409ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.056+0000 7fcaaa3e7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcaa4073fb0 0x7fcaa40b4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.056+0000 7fcaaa3e7640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaa415f7a0 con 0x7fcaa40b4b60 2024-05-09T05:01:44.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.056+0000 7fcaaa3e7640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fcaa4071cb0 con 0x7fcaa415f3c0 2024-05-09T05:01:44.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.056+0000 7fcaaa3e7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fcaa409f480 con 0x7fcaa4073fb0 2024-05-09T05:01:44.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.056+0000 7fcaa3fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcaa415f3c0 0x7fcaa409ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.056+0000 7fcaa8984640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcaa40b4b60 0x7fcaa406e770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57766/0 (socket says 172.21.15.60:57766) 2024-05-09T05:01:44.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.056+0000 7fcaa8984640 1 -- 172.21.15.60:0/1700554310 learned_addr learned my addr 172.21.15.60:0/1700554310 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:44.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.056+0000 7fcaa37fe640 1 --2- 172.21.15.60:0/1700554310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcaa4073fb0 0x7fcaa40b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.056+0000 7fcaa3fff640 1 -- 172.21.15.60:0/1700554310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcaa4073fb0 msgr2=0x7fcaa40b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:44.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.056+0000 7fcaa3fff640 1 --2- 172.21.15.60:0/1700554310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcaa4073fb0 0x7fcaa40b4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.056+0000 7fcaa3fff640 1 -- 172.21.15.60:0/1700554310 >> v1:172.21.15.60:6789/0 conn(0x7fcaa40b4b60 legacy=0x7fcaa406e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:44.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.056+0000 7fcaa3fff640 1 -- 172.21.15.60:0/1700554310 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaa409f760 con 0x7fcaa415f3c0 2024-05-09T05:01:44.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.056+0000 7fcaa2ffd640 1 -- 172.21.15.60:0/1700554310 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 378008065 0 0) 0x7fcaa415f7a0 con 0x7fcaa40b4b60 2024-05-09T05:01:44.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.057+0000 7fcaa3fff640 1 --2- 172.21.15.60:0/1700554310 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcaa415f3c0 0x7fcaa409ebe0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fca940097a0 tx=0x7fca9402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7053f997f100a967 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:44.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.057+0000 7fcaa2ffd640 1 -- 172.21.15.60:0/1700554310 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca94037070 con 0x7fcaa415f3c0 2024-05-09T05:01:44.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.057+0000 7fcaa2ffd640 1 -- 172.21.15.60:0/1700554310 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca9402ae50 con 0x7fcaa415f3c0 2024-05-09T05:01:44.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.057+0000 7fcaa2ffd640 1 -- 172.21.15.60:0/1700554310 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca94032660 con 0x7fcaa415f3c0 2024-05-09T05:01:44.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.058+0000 7fcaaa3e7640 1 -- 172.21.15.60:0/1700554310 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcaa415f3c0 msgr2=0x7fcaa409ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:44.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.058+0000 7fcaaa3e7640 1 --2- 172.21.15.60:0/1700554310 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcaa415f3c0 0x7fcaa409ebe0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fca940097a0 tx=0x7fca9402a100 comp rx=0 tx=0).stop 2024-05-09T05:01:44.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.058+0000 7fcaaa3e7640 1 -- 172.21.15.60:0/1700554310 shutdown_connections 2024-05-09T05:01:44.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.058+0000 7fcaaa3e7640 1 --2- 172.21.15.60:0/1700554310 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcaa4073fb0 0x7fcaa40b4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.058+0000 7fcaaa3e7640 1 --2- 172.21.15.60:0/1700554310 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcaa415f3c0 0x7fcaa409ebe0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.058+0000 7fcaaa3e7640 1 -- 172.21.15.60:0/1700554310 >> 172.21.15.60:0/1700554310 conn(0x7fcaa40d1470 msgr2=0x7fcaa40d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:44.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.058+0000 7fcaaa3e7640 1 -- 172.21.15.60:0/1700554310 shutdown_connections 2024-05-09T05:01:44.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.058+0000 7fcaaa3e7640 1 -- 172.21.15.60:0/1700554310 wait complete. 2024-05-09T05:01:44.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.059+0000 7fcaaa3e7640 1 Processor -- start 2024-05-09T05:01:44.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.059+0000 7fcaaa3e7640 1 -- start start 2024-05-09T05:01:44.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.060+0000 7fcaaa3e7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcaa4073fb0 0x7fcaa415e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.060+0000 7fcaaa3e7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcaa40b4b60 0x7fcaa40c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.060+0000 7fcaaa3e7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcaa415f3c0 0x7fcaa40c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.060+0000 7fcaaa3e7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa4073150 con 0x7fcaa40b4b60 2024-05-09T05:01:44.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.060+0000 7fcaaa3e7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa409f120 con 0x7fcaa415f3c0 2024-05-09T05:01:44.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.060+0000 7fcaaa3e7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fcaa40732d0 con 0x7fcaa4073fb0 2024-05-09T05:01:44.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.060+0000 7fcaa37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcaa40b4b60 0x7fcaa40c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.060+0000 7fcaa37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcaa40b4b60 0x7fcaa40c90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37740/0 (socket says 172.21.15.60:37740) 2024-05-09T05:01:44.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.060+0000 7fcaa37fe640 1 -- 172.21.15.60:0/2160462776 learned_addr learned my addr 172.21.15.60:0/2160462776 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:44.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.060+0000 7fcaa3fff640 1 --2- 172.21.15.60:0/2160462776 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcaa4073fb0 0x7fcaa415e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.060+0000 7fcaa37fe640 1 -- 172.21.15.60:0/2160462776 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcaa4073fb0 msgr2=0x7fcaa415e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:44.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.060+0000 7fcaa8984640 1 --2- 172.21.15.60:0/2160462776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcaa415f3c0 0x7fcaa40c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.060+0000 7fcaa37fe640 1 --2- 172.21.15.60:0/2160462776 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcaa4073fb0 0x7fcaa415e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.060+0000 7fcaa37fe640 1 -- 172.21.15.60:0/2160462776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcaa415f3c0 msgr2=0x7fcaa40c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:44.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.060+0000 7fcaa37fe640 1 --2- 172.21.15.60:0/2160462776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcaa415f3c0 0x7fcaa40c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.060+0000 7fcaa37fe640 1 -- 172.21.15.60:0/2160462776 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaa40c9b20 con 0x7fcaa40b4b60 2024-05-09T05:01:44.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.060+0000 7fcaa3fff640 1 --2- 172.21.15.60:0/2160462776 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcaa4073fb0 0x7fcaa415e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:44.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.060+0000 7fcaa8984640 1 --2- 172.21.15.60:0/2160462776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcaa415f3c0 0x7fcaa40c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:44.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.060+0000 7fcaa37fe640 1 --2- 172.21.15.60:0/2160462776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcaa40b4b60 0x7fcaa40c90a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fca9800e9e0 tx=0x7fca9800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:44.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.061+0000 7fcaa0ff9640 1 -- 172.21.15.60:0/2160462776 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca98017070 con 0x7fcaa40b4b60 2024-05-09T05:01:44.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.061+0000 7fcaa0ff9640 1 -- 172.21.15.60:0/2160462776 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca9800c9d0 con 0x7fcaa40b4b60 2024-05-09T05:01:44.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.061+0000 7fcaa0ff9640 1 -- 172.21.15.60:0/2160462776 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca9800cdc0 con 0x7fcaa40b4b60 2024-05-09T05:01:44.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.061+0000 7fcaaa3e7640 1 -- 172.21.15.60:0/2160462776 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaa40d4490 con 0x7fcaa40b4b60 2024-05-09T05:01:44.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.061+0000 7fcaaa3e7640 1 -- 172.21.15.60:0/2160462776 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaa40d4a50 con 0x7fcaa40b4b60 2024-05-09T05:01:44.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.062+0000 7fcaaa3e7640 1 -- 172.21.15.60:0/2160462776 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaa40d4e50 con 0x7fcaa40b4b60 2024-05-09T05:01:44.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.062+0000 7fcaa0ff9640 1 -- 172.21.15.60:0/2160462776 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fca9801a070 con 0x7fcaa40b4b60 2024-05-09T05:01:44.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.063+0000 7fcaa0ff9640 1 --2- 172.21.15.60:0/2160462776 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fca900762d0 0x7fca90078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.063+0000 7fcaa0ff9640 1 -- 172.21.15.60:0/2160462776 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fca98097860 con 0x7fcaa40b4b60 2024-05-09T05:01:44.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.063+0000 7fcaa3fff640 1 --2- 172.21.15.60:0/2160462776 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fca900762d0 0x7fca90078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.064+0000 7fcaa3fff640 1 --2- 172.21.15.60:0/2160462776 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fca900762d0 0x7fca90078790 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fca94034000 tx=0x7fca9402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:44.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.067+0000 7fcaa0ff9640 1 -- 172.21.15.60:0/2160462776 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fcaa40d4e50 con 0x7fcaa40b4b60 2024-05-09T05:01:44.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.254+0000 7fcaaa3e7640 1 -- 172.21.15.60:0/2160462776 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcaa41f4cc0 con 0x7fca900762d0 2024-05-09T05:01:44.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.260+0000 7fcaa0ff9640 1 -- 172.21.15.60:0/2160462776 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251942 (secure 0 0 0) 0x7fcaa41f4cc0 con 0x7fca900762d0 2024-05-09T05:01:44.263 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:44.264 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-05-09T05:01:44.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.263+0000 7fcaaa3e7640 1 -- 172.21.15.60:0/2160462776 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fca900762d0 msgr2=0x7fca90078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:44.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.263+0000 7fcaaa3e7640 1 --2- 172.21.15.60:0/2160462776 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fca900762d0 0x7fca90078790 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fca94034000 tx=0x7fca9402e000 comp rx=0 tx=0).stop 2024-05-09T05:01:44.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.263+0000 7fcaaa3e7640 1 -- 172.21.15.60:0/2160462776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcaa40b4b60 msgr2=0x7fcaa40c90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:44.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.263+0000 7fcaaa3e7640 1 --2- 172.21.15.60:0/2160462776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcaa40b4b60 0x7fcaa40c90a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fca9800e9e0 tx=0x7fca9800eea0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.263+0000 7fcaaa3e7640 1 -- 172.21.15.60:0/2160462776 shutdown_connections 2024-05-09T05:01:44.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.263+0000 7fcaaa3e7640 1 --2- 172.21.15.60:0/2160462776 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fca900762d0 0x7fca90078790 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.263+0000 7fcaaa3e7640 1 --2- 172.21.15.60:0/2160462776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcaa415f3c0 0x7fcaa40c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.263+0000 7fcaaa3e7640 1 --2- 172.21.15.60:0/2160462776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcaa40b4b60 0x7fcaa40c90a0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.263+0000 7fcaaa3e7640 1 --2- 172.21.15.60:0/2160462776 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fcaa4073fb0 0x7fcaa415e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.263+0000 7fcaaa3e7640 1 -- 172.21.15.60:0/2160462776 >> 172.21.15.60:0/2160462776 conn(0x7fcaa40d1470 msgr2=0x7fcaa4072140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:44.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.263+0000 7fcaaa3e7640 1 -- 172.21.15.60:0/2160462776 shutdown_connections 2024-05-09T05:01:44.267 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.263+0000 7fcaaa3e7640 1 -- 172.21.15.60:0/2160462776 wait complete. 2024-05-09T05:01:44.275 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-05-09T05:01:42.930309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238792,"kb_used_data":5968,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735928,"statfs":{"total":773094113280,"available":772849590272,"internally_reserved":0,"allocated":6111232,"data_stored":3341700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.084541"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.852002+0000","last_change":"2024-05-09T05:01:39.875218+0000","last_active":"2024-05-09T05:01:40.852002+0000","last_peered":"2024-05-09T05:01:40.852002+0000","last_clean":"2024-05-09T05:01:40.852002+0000","last_became_active":"2024-05-09T05:01:39.874925+0000","last_became_peered":"2024-05-09T05:01:39.874925+0000","last_unstale":"2024-05-09T05:01:40.852002+0000","last_undegraded":"2024-05-09T05:01:40.852002+0000","last_fullsized":"2024-05-09T05:01:40.852002+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:12.625336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851053+0000","last_change":"2024-05-09T05:01:39.864266+0000","last_active":"2024-05-09T05:01:40.851053+0000","last_peered":"2024-05-09T05:01:40.851053+0000","last_clean":"2024-05-09T05:01:40.851053+0000","last_became_active":"2024-05-09T05:01:39.863965+0000","last_became_peered":"2024-05-09T05:01:39.863965+0000","last_unstale":"2024-05-09T05:01:40.851053+0000","last_undegraded":"2024-05-09T05:01:40.851053+0000","last_fullsized":"2024-05-09T05:01:40.851053+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:01:57.964834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857444+0000","last_change":"2024-05-09T05:01:39.867479+0000","last_active":"2024-05-09T05:01:40.857444+0000","last_peered":"2024-05-09T05:01:40.857444+0000","last_clean":"2024-05-09T05:01:40.857444+0000","last_became_active":"2024-05-09T05:01:39.867215+0000","last_became_peered":"2024-05-09T05:01:39.867215+0000","last_unstale":"2024-05-09T05:01:40.857444+0000","last_undegraded":"2024-05-09T05:01:40.857444+0000","last_fullsized":"2024-05-09T05:01:40.857444+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:57:48.604326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358390+0000","last_change":"2024-05-09T05:01:39.864540+0000","last_active":"2024-05-09T05:01:41.358390+0000","last_peered":"2024-05-09T05:01:41.358390+0000","last_clean":"2024-05-09T05:01:41.358390+0000","last_became_active":"2024-05-09T05:01:39.864203+0000","last_became_peered":"2024-05-09T05:01:39.864203+0000","last_unstale":"2024-05-09T05:01:41.358390+0000","last_undegraded":"2024-05-09T05:01:41.358390+0000","last_fullsized":"2024-05-09T05:01:41.358390+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:48:03.359391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851847+0000","last_change":"2024-05-09T05:01:39.871306+0000","last_active":"2024-05-09T05:01:40.851847+0000","last_peered":"2024-05-09T05:01:40.851847+0000","last_clean":"2024-05-09T05:01:40.851847+0000","last_became_active":"2024-05-09T05:01:39.870998+0000","last_became_peered":"2024-05-09T05:01:39.870998+0000","last_unstale":"2024-05-09T05:01:40.851847+0000","last_undegraded":"2024-05-09T05:01:40.851847+0000","last_fullsized":"2024-05-09T05:01:40.851847+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:38.550789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358285+0000","last_change":"2024-05-09T05:01:39.875924+0000","last_active":"2024-05-09T05:01:41.358285+0000","last_peered":"2024-05-09T05:01:41.358285+0000","last_clean":"2024-05-09T05:01:41.358285+0000","last_became_active":"2024-05-09T05:01:39.875178+0000","last_became_peered":"2024-05-09T05:01:39.875178+0000","last_unstale":"2024-05-09T05:01:41.358285+0000","last_undegraded":"2024-05-09T05:01:41.358285+0000","last_fullsized":"2024-05-09T05:01:41.358285+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:37:52.332218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980355+0000","last_change":"2024-05-09T05:01:39.870489+0000","last_active":"2024-05-09T05:01:40.980355+0000","last_peered":"2024-05-09T05:01:40.980355+0000","last_clean":"2024-05-09T05:01:40.980355+0000","last_became_active":"2024-05-09T05:01:39.868433+0000","last_became_peered":"2024-05-09T05:01:39.868433+0000","last_unstale":"2024-05-09T05:01:40.980355+0000","last_undegraded":"2024-05-09T05:01:40.980355+0000","last_fullsized":"2024-05-09T05:01:40.980355+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:06.258896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.850921+0000","last_change":"2024-05-09T05:01:39.854222+0000","last_active":"2024-05-09T05:01:40.850921+0000","last_peered":"2024-05-09T05:01:40.850921+0000","last_clean":"2024-05-09T05:01:40.850921+0000","last_became_active":"2024-05-09T05:01:39.853927+0000","last_became_peered":"2024-05-09T05:01:39.853927+0000","last_unstale":"2024-05-09T05:01:40.850921+0000","last_undegraded":"2024-05-09T05:01:40.850921+0000","last_fullsized":"2024-05-09T05:01:40.850921+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:41:31.058309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980674+0000","last_change":"2024-05-09T05:01:39.861561+0000","last_active":"2024-05-09T05:01:40.980674+0000","last_peered":"2024-05-09T05:01:40.980674+0000","last_clean":"2024-05-09T05:01:40.980674+0000","last_became_active":"2024-05-09T05:01:39.861026+0000","last_became_peered":"2024-05-09T05:01:39.861026+0000","last_unstale":"2024-05-09T05:01:40.980674+0000","last_undegraded":"2024-05-09T05:01:40.980674+0000","last_fullsized":"2024-05-09T05:01:40.980674+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:16:28.592034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358930+0000","last_change":"2024-05-09T05:01:39.866566+0000","last_active":"2024-05-09T05:01:41.358930+0000","last_peered":"2024-05-09T05:01:41.358930+0000","last_clean":"2024-05-09T05:01:41.358930+0000","last_became_active":"2024-05-09T05:01:39.866315+0000","last_became_peered":"2024-05-09T05:01:39.866315+0000","last_unstale":"2024-05-09T05:01:41.358930+0000","last_undegraded":"2024-05-09T05:01:41.358930+0000","last_fullsized":"2024-05-09T05:01:41.358930+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:05:22.823683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358777+0000","last_change":"2024-05-09T05:01:39.869309+0000","last_active":"2024-05-09T05:01:41.358777+0000","last_peered":"2024-05-09T05:01:41.358777+0000","last_clean":"2024-05-09T05:01:41.358777+0000","last_became_active":"2024-05-09T05:01:39.868934+0000","last_became_peered":"2024-05-09T05:01:39.868934+0000","last_unstale":"2024-05-09T05:01:41.358777+0000","last_undegraded":"2024-05-09T05:01:41.358777+0000","last_fullsized":"2024-05-09T05:01:41.358777+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:47:16.952818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358583+0000","last_change":"2024-05-09T05:01:39.873199+0000","last_active":"2024-05-09T05:01:41.358583+0000","last_peered":"2024-05-09T05:01:41.358583+0000","last_clean":"2024-05-09T05:01:41.358583+0000","last_became_active":"2024-05-09T05:01:39.872903+0000","last_became_peered":"2024-05-09T05:01:39.872903+0000","last_unstale":"2024-05-09T05:01:41.358583+0000","last_undegraded":"2024-05-09T05:01:41.358583+0000","last_fullsized":"2024-05-09T05:01:41.358583+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:17:38.783370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358286+0000","last_change":"2024-05-09T05:01:39.874859+0000","last_active":"2024-05-09T05:01:41.358286+0000","last_peered":"2024-05-09T05:01:41.358286+0000","last_clean":"2024-05-09T05:01:41.358286+0000","last_became_active":"2024-05-09T05:01:39.874533+0000","last_became_peered":"2024-05-09T05:01:39.874533+0000","last_unstale":"2024-05-09T05:01:41.358286+0000","last_undegraded":"2024-05-09T05:01:41.358286+0000","last_fullsized":"2024-05-09T05:01:41.358286+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:35.258317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980624+0000","last_change":"2024-05-09T05:01:39.865170+0000","last_active":"2024-05-09T05:01:40.980624+0000","last_peered":"2024-05-09T05:01:40.980624+0000","last_clean":"2024-05-09T05:01:40.980624+0000","last_became_active":"2024-05-09T05:01:39.864867+0000","last_became_peered":"2024-05-09T05:01:39.864867+0000","last_unstale":"2024-05-09T05:01:40.980624+0000","last_undegraded":"2024-05-09T05:01:40.980624+0000","last_fullsized":"2024-05-09T05:01:40.980624+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:21:29.020025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980606+0000","last_change":"2024-05-09T05:01:39.868574+0000","last_active":"2024-05-09T05:01:40.980606+0000","last_peered":"2024-05-09T05:01:40.980606+0000","last_clean":"2024-05-09T05:01:40.980606+0000","last_became_active":"2024-05-09T05:01:39.865956+0000","last_became_peered":"2024-05-09T05:01:39.865956+0000","last_unstale":"2024-05-09T05:01:40.980606+0000","last_undegraded":"2024-05-09T05:01:40.980606+0000","last_fullsized":"2024-05-09T05:01:40.980606+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:27:07.564385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358004+0000","last_change":"2024-05-09T05:01:39.867547+0000","last_active":"2024-05-09T05:01:41.358004+0000","last_peered":"2024-05-09T05:01:41.358004+0000","last_clean":"2024-05-09T05:01:41.358004+0000","last_became_active":"2024-05-09T05:01:39.867266+0000","last_became_peered":"2024-05-09T05:01:39.867266+0000","last_unstale":"2024-05-09T05:01:41.358004+0000","last_undegraded":"2024-05-09T05:01:41.358004+0000","last_fullsized":"2024-05-09T05:01:41.358004+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:24.553056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851453+0000","last_change":"2024-05-09T05:01:39.872382+0000","last_active":"2024-05-09T05:01:40.851453+0000","last_peered":"2024-05-09T05:01:40.851453+0000","last_clean":"2024-05-09T05:01:40.851453+0000","last_became_active":"2024-05-09T05:01:39.872055+0000","last_became_peered":"2024-05-09T05:01:39.872055+0000","last_unstale":"2024-05-09T05:01:40.851453+0000","last_undegraded":"2024-05-09T05:01:40.851453+0000","last_fullsized":"2024-05-09T05:01:40.851453+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:34:58.245332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857667+0000","last_change":"2024-05-09T05:01:39.863769+0000","last_active":"2024-05-09T05:01:40.857667+0000","last_peered":"2024-05-09T05:01:40.857667+0000","last_clean":"2024-05-09T05:01:40.857667+0000","last_became_active":"2024-05-09T05:01:39.863396+0000","last_became_peered":"2024-05-09T05:01:39.863396+0000","last_unstale":"2024-05-09T05:01:40.857667+0000","last_undegraded":"2024-05-09T05:01:40.857667+0000","last_fullsized":"2024-05-09T05:01:40.857667+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:02:53.110445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851562+0000","last_change":"2024-05-09T05:01:39.875936+0000","last_active":"2024-05-09T05:01:40.851562+0000","last_peered":"2024-05-09T05:01:40.851562+0000","last_clean":"2024-05-09T05:01:40.851562+0000","last_became_active":"2024-05-09T05:01:39.875614+0000","last_became_peered":"2024-05-09T05:01:39.875614+0000","last_unstale":"2024-05-09T05:01:40.851562+0000","last_undegraded":"2024-05-09T05:01:40.851562+0000","last_fullsized":"2024-05-09T05:01:40.851562+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:30.745037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851207+0000","last_change":"2024-05-09T05:01:39.867227+0000","last_active":"2024-05-09T05:01:40.851207+0000","last_peered":"2024-05-09T05:01:40.851207+0000","last_clean":"2024-05-09T05:01:40.851207+0000","last_became_active":"2024-05-09T05:01:39.866882+0000","last_became_peered":"2024-05-09T05:01:39.866882+0000","last_unstale":"2024-05-09T05:01:40.851207+0000","last_undegraded":"2024-05-09T05:01:40.851207+0000","last_fullsized":"2024-05-09T05:01:40.851207+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:01:16.557587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857353+0000","last_change":"2024-05-09T05:01:39.872958+0000","last_active":"2024-05-09T05:01:40.857353+0000","last_peered":"2024-05-09T05:01:40.857353+0000","last_clean":"2024-05-09T05:01:40.857353+0000","last_became_active":"2024-05-09T05:01:39.872567+0000","last_became_peered":"2024-05-09T05:01:39.872567+0000","last_unstale":"2024-05-09T05:01:40.857353+0000","last_undegraded":"2024-05-09T05:01:40.857353+0000","last_fullsized":"2024-05-09T05:01:40.857353+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:21.336536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851512+0000","last_change":"2024-05-09T05:01:39.873730+0000","last_active":"2024-05-09T05:01:40.851512+0000","last_peered":"2024-05-09T05:01:40.851512+0000","last_clean":"2024-05-09T05:01:40.851512+0000","last_became_active":"2024-05-09T05:01:39.873271+0000","last_became_peered":"2024-05-09T05:01:39.873271+0000","last_unstale":"2024-05-09T05:01:40.851512+0000","last_undegraded":"2024-05-09T05:01:40.851512+0000","last_fullsized":"2024-05-09T05:01:40.851512+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:51:42.545363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857303+0000","last_change":"2024-05-09T05:01:39.873510+0000","last_active":"2024-05-09T05:01:40.857303+0000","last_peered":"2024-05-09T05:01:40.857303+0000","last_clean":"2024-05-09T05:01:40.857303+0000","last_became_active":"2024-05-09T05:01:39.873226+0000","last_became_peered":"2024-05-09T05:01:39.873226+0000","last_unstale":"2024-05-09T05:01:40.857303+0000","last_undegraded":"2024-05-09T05:01:40.857303+0000","last_fullsized":"2024-05-09T05:01:40.857303+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:37:52.332218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.357859+0000","last_change":"2024-05-09T05:01:39.866948+0000","last_active":"2024-05-09T05:01:41.357859+0000","last_peered":"2024-05-09T05:01:41.357859+0000","last_clean":"2024-05-09T05:01:41.357859+0000","last_became_active":"2024-05-09T05:01:39.866329+0000","last_became_peered":"2024-05-09T05:01:39.866329+0000","last_unstale":"2024-05-09T05:01:41.357859+0000","last_undegraded":"2024-05-09T05:01:41.357859+0000","last_fullsized":"2024-05-09T05:01:41.357859+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:06.258896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.852139+0000","last_change":"2024-05-09T05:01:39.880371+0000","last_active":"2024-05-09T05:01:40.852139+0000","last_peered":"2024-05-09T05:01:40.852139+0000","last_clean":"2024-05-09T05:01:40.852139+0000","last_became_active":"2024-05-09T05:01:39.880114+0000","last_became_peered":"2024-05-09T05:01:39.880114+0000","last_unstale":"2024-05-09T05:01:40.852139+0000","last_undegraded":"2024-05-09T05:01:40.852139+0000","last_fullsized":"2024-05-09T05:01:40.852139+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:07:02.245459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358343+0000","last_change":"2024-05-09T05:01:39.871088+0000","last_active":"2024-05-09T05:01:41.358343+0000","last_peered":"2024-05-09T05:01:41.358343+0000","last_clean":"2024-05-09T05:01:41.358343+0000","last_became_active":"2024-05-09T05:01:39.870806+0000","last_became_peered":"2024-05-09T05:01:39.870806+0000","last_unstale":"2024-05-09T05:01:41.358343+0000","last_undegraded":"2024-05-09T05:01:41.358343+0000","last_fullsized":"2024-05-09T05:01:41.358343+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:57:48.604326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358286+0000","last_change":"2024-05-09T05:01:39.865930+0000","last_active":"2024-05-09T05:01:41.358286+0000","last_peered":"2024-05-09T05:01:41.358286+0000","last_clean":"2024-05-09T05:01:41.358286+0000","last_became_active":"2024-05-09T05:01:39.865689+0000","last_became_peered":"2024-05-09T05:01:39.865689+0000","last_unstale":"2024-05-09T05:01:41.358286+0000","last_undegraded":"2024-05-09T05:01:41.358286+0000","last_fullsized":"2024-05-09T05:01:41.358286+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:59:30.203215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.852182+0000","last_change":"2024-05-09T05:01:39.868732+0000","last_active":"2024-05-09T05:01:40.852182+0000","last_peered":"2024-05-09T05:01:40.852182+0000","last_clean":"2024-05-09T05:01:40.852182+0000","last_became_active":"2024-05-09T05:01:39.868433+0000","last_became_peered":"2024-05-09T05:01:39.868433+0000","last_unstale":"2024-05-09T05:01:40.852182+0000","last_undegraded":"2024-05-09T05:01:40.852182+0000","last_fullsized":"2024-05-09T05:01:40.852182+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:20:33.314890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851296+0000","last_change":"2024-05-09T05:01:39.876197+0000","last_active":"2024-05-09T05:01:40.851296+0000","last_peered":"2024-05-09T05:01:40.851296+0000","last_clean":"2024-05-09T05:01:40.851296+0000","last_became_active":"2024-05-09T05:01:39.875857+0000","last_became_peered":"2024-05-09T05:01:39.875857+0000","last_unstale":"2024-05-09T05:01:40.851296+0000","last_undegraded":"2024-05-09T05:01:40.851296+0000","last_fullsized":"2024-05-09T05:01:40.851296+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:42:40.603692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857399+0000","last_change":"2024-05-09T05:01:39.862165+0000","last_active":"2024-05-09T05:01:40.857399+0000","last_peered":"2024-05-09T05:01:40.857399+0000","last_clean":"2024-05-09T05:01:40.857399+0000","last_became_active":"2024-05-09T05:01:39.861827+0000","last_became_peered":"2024-05-09T05:01:39.861827+0000","last_unstale":"2024-05-09T05:01:40.857399+0000","last_undegraded":"2024-05-09T05:01:40.857399+0000","last_fullsized":"2024-05-09T05:01:40.857399+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:06.942055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851402+0000","last_change":"2024-05-09T05:01:39.878415+0000","last_active":"2024-05-09T05:01:40.851402+0000","last_peered":"2024-05-09T05:01:40.851402+0000","last_clean":"2024-05-09T05:01:40.851402+0000","last_became_active":"2024-05-09T05:01:39.878000+0000","last_became_peered":"2024-05-09T05:01:39.878000+0000","last_unstale":"2024-05-09T05:01:40.851402+0000","last_undegraded":"2024-05-09T05:01:40.851402+0000","last_fullsized":"2024-05-09T05:01:40.851402+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:07:34.624914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358065+0000","last_change":"2024-05-09T05:01:39.882587+0000","last_active":"2024-05-09T05:01:41.358065+0000","last_peered":"2024-05-09T05:01:41.358065+0000","last_clean":"2024-05-09T05:01:41.358065+0000","last_became_active":"2024-05-09T05:01:39.880797+0000","last_became_peered":"2024-05-09T05:01:39.880797+0000","last_unstale":"2024-05-09T05:01:41.358065+0000","last_undegraded":"2024-05-09T05:01:41.358065+0000","last_fullsized":"2024-05-09T05:01:41.358065+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:16:55.605416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358313+0000","last_change":"2024-05-09T05:01:39.873638+0000","last_active":"2024-05-09T05:01:41.358313+0000","last_peered":"2024-05-09T05:01:41.358313+0000","last_clean":"2024-05-09T05:01:41.358313+0000","last_became_active":"2024-05-09T05:01:39.873367+0000","last_became_peered":"2024-05-09T05:01:39.873367+0000","last_unstale":"2024-05-09T05:01:41.358313+0000","last_undegraded":"2024-05-09T05:01:41.358313+0000","last_fullsized":"2024-05-09T05:01:41.358313+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:21.336536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"50'4","reported_seq":18,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358595+0000","last_change":"2024-05-09T05:01:38.861502+0000","last_active":"2024-05-09T05:01:41.358595+0000","last_peered":"2024-05-09T05:01:41.358595+0000","last_clean":"2024-05-09T05:01:41.358595+0000","last_became_active":"2024-05-09T05:01:38.861131+0000","last_became_peered":"2024-05-09T05:01:38.861131+0000","last_unstale":"2024-05-09T05:01:41.358595+0000","last_undegraded":"2024-05-09T05:01:41.358595+0000","last_fullsized":"2024-05-09T05:01:41.358595+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:47:15.947904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358929+0000","last_change":"2024-05-09T05:01:39.878999+0000","last_active":"2024-05-09T05:01:41.358929+0000","last_peered":"2024-05-09T05:01:41.358929+0000","last_clean":"2024-05-09T05:01:41.358929+0000","last_became_active":"2024-05-09T05:01:39.874591+0000","last_became_peered":"2024-05-09T05:01:39.874591+0000","last_unstale":"2024-05-09T05:01:41.358929+0000","last_undegraded":"2024-05-09T05:01:41.358929+0000","last_fullsized":"2024-05-09T05:01:41.358929+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:07:16.662667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851161+0000","last_change":"2024-05-09T05:01:38.862945+0000","last_active":"2024-05-09T05:01:40.851161+0000","last_peered":"2024-05-09T05:01:40.851161+0000","last_clean":"2024-05-09T05:01:40.851161+0000","last_became_active":"2024-05-09T05:01:38.862633+0000","last_became_peered":"2024-05-09T05:01:38.862633+0000","last_unstale":"2024-05-09T05:01:40.851161+0000","last_undegraded":"2024-05-09T05:01:40.851161+0000","last_fullsized":"2024-05-09T05:01:40.851161+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:39:03.448180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358625+0000","last_change":"2024-05-09T05:01:39.862888+0000","last_active":"2024-05-09T05:01:41.358625+0000","last_peered":"2024-05-09T05:01:41.358625+0000","last_clean":"2024-05-09T05:01:41.358625+0000","last_became_active":"2024-05-09T05:01:39.862585+0000","last_became_peered":"2024-05-09T05:01:39.862585+0000","last_unstale":"2024-05-09T05:01:41.358625+0000","last_undegraded":"2024-05-09T05:01:41.358625+0000","last_fullsized":"2024-05-09T05:01:41.358625+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:26.273557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358718+0000","last_change":"2024-05-09T05:01:38.857444+0000","last_active":"2024-05-09T05:01:41.358718+0000","last_peered":"2024-05-09T05:01:41.358718+0000","last_clean":"2024-05-09T05:01:41.358718+0000","last_became_active":"2024-05-09T05:01:38.857133+0000","last_became_peered":"2024-05-09T05:01:38.857133+0000","last_unstale":"2024-05-09T05:01:41.358718+0000","last_undegraded":"2024-05-09T05:01:41.358718+0000","last_fullsized":"2024-05-09T05:01:41.358718+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:39:57.155429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358464+0000","last_change":"2024-05-09T05:01:39.870882+0000","last_active":"2024-05-09T05:01:41.358464+0000","last_peered":"2024-05-09T05:01:41.358464+0000","last_clean":"2024-05-09T05:01:41.358464+0000","last_became_active":"2024-05-09T05:01:39.870567+0000","last_became_peered":"2024-05-09T05:01:39.870567+0000","last_unstale":"2024-05-09T05:01:41.358464+0000","last_undegraded":"2024-05-09T05:01:41.358464+0000","last_fullsized":"2024-05-09T05:01:41.358464+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:21.186391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358528+0000","last_change":"2024-05-09T05:01:38.863364+0000","last_active":"2024-05-09T05:01:41.358528+0000","last_peered":"2024-05-09T05:01:41.358528+0000","last_clean":"2024-05-09T05:01:41.358528+0000","last_became_active":"2024-05-09T05:01:38.863031+0000","last_became_peered":"2024-05-09T05:01:38.863031+0000","last_unstale":"2024-05-09T05:01:41.358528+0000","last_undegraded":"2024-05-09T05:01:41.358528+0000","last_fullsized":"2024-05-09T05:01:41.358528+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:53:39.994209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.850817+0000","last_change":"2024-05-09T05:01:39.873520+0000","last_active":"2024-05-09T05:01:40.850817+0000","last_peered":"2024-05-09T05:01:40.850817+0000","last_clean":"2024-05-09T05:01:40.850817+0000","last_became_active":"2024-05-09T05:01:39.872905+0000","last_became_peered":"2024-05-09T05:01:39.872905+0000","last_unstale":"2024-05-09T05:01:40.850817+0000","last_undegraded":"2024-05-09T05:01:40.850817+0000","last_fullsized":"2024-05-09T05:01:40.850817+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:37:47.075587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980479+0000","last_change":"2024-05-09T05:01:38.862527+0000","last_active":"2024-05-09T05:01:40.980479+0000","last_peered":"2024-05-09T05:01:40.980479+0000","last_clean":"2024-05-09T05:01:40.980479+0000","last_became_active":"2024-05-09T05:01:38.861979+0000","last_became_peered":"2024-05-09T05:01:38.861979+0000","last_unstale":"2024-05-09T05:01:40.980479+0000","last_undegraded":"2024-05-09T05:01:40.980479+0000","last_fullsized":"2024-05-09T05:01:40.980479+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:31:57.936029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851068+0000","last_change":"2024-05-09T05:01:39.865218+0000","last_active":"2024-05-09T05:01:40.851068+0000","last_peered":"2024-05-09T05:01:40.851068+0000","last_clean":"2024-05-09T05:01:40.851068+0000","last_became_active":"2024-05-09T05:01:39.864965+0000","last_became_peered":"2024-05-09T05:01:39.864965+0000","last_unstale":"2024-05-09T05:01:40.851068+0000","last_undegraded":"2024-05-09T05:01:40.851068+0000","last_fullsized":"2024-05-09T05:01:40.851068+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:50:38.445530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358239+0000","last_change":"2024-05-09T05:01:38.865608+0000","last_active":"2024-05-09T05:01:41.358239+0000","last_peered":"2024-05-09T05:01:41.358239+0000","last_clean":"2024-05-09T05:01:41.358239+0000","last_became_active":"2024-05-09T05:01:38.865155+0000","last_became_peered":"2024-05-09T05:01:38.865155+0000","last_unstale":"2024-05-09T05:01:41.358239+0000","last_undegraded":"2024-05-09T05:01:41.358239+0000","last_fullsized":"2024-05-09T05:01:41.358239+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:22:23.371054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.357800+0000","last_change":"2024-05-09T05:01:39.862637+0000","last_active":"2024-05-09T05:01:41.357800+0000","last_peered":"2024-05-09T05:01:41.357800+0000","last_clean":"2024-05-09T05:01:41.357800+0000","last_became_active":"2024-05-09T05:01:39.862059+0000","last_became_peered":"2024-05-09T05:01:39.862059+0000","last_unstale":"2024-05-09T05:01:41.357800+0000","last_undegraded":"2024-05-09T05:01:41.357800+0000","last_fullsized":"2024-05-09T05:01:41.357800+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:51:43.777982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358236+0000","last_change":"2024-05-09T05:01:38.865473+0000","last_active":"2024-05-09T05:01:41.358236+0000","last_peered":"2024-05-09T05:01:41.358236+0000","last_clean":"2024-05-09T05:01:41.358236+0000","last_became_active":"2024-05-09T05:01:38.864641+0000","last_became_peered":"2024-05-09T05:01:38.864641+0000","last_unstale":"2024-05-09T05:01:41.358236+0000","last_undegraded":"2024-05-09T05:01:41.358236+0000","last_fullsized":"2024-05-09T05:01:41.358236+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:39.777738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.850858+0000","last_change":"2024-05-09T05:01:39.871696+0000","last_active":"2024-05-09T05:01:40.850858+0000","last_peered":"2024-05-09T05:01:40.850858+0000","last_clean":"2024-05-09T05:01:40.850858+0000","last_became_active":"2024-05-09T05:01:39.871423+0000","last_became_peered":"2024-05-09T05:01:39.871423+0000","last_unstale":"2024-05-09T05:01:40.850858+0000","last_undegraded":"2024-05-09T05:01:40.850858+0000","last_fullsized":"2024-05-09T05:01:40.850858+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:03:16.895228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980459+0000","last_change":"2024-05-09T05:01:38.866604+0000","last_active":"2024-05-09T05:01:40.980459+0000","last_peered":"2024-05-09T05:01:40.980459+0000","last_clean":"2024-05-09T05:01:40.980459+0000","last_became_active":"2024-05-09T05:01:38.866324+0000","last_became_peered":"2024-05-09T05:01:38.866324+0000","last_unstale":"2024-05-09T05:01:40.980459+0000","last_undegraded":"2024-05-09T05:01:40.980459+0000","last_fullsized":"2024-05-09T05:01:40.980459+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:09:22.170335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980784+0000","last_change":"2024-05-09T05:01:39.866146+0000","last_active":"2024-05-09T05:01:40.980784+0000","last_peered":"2024-05-09T05:01:40.980784+0000","last_clean":"2024-05-09T05:01:40.980784+0000","last_became_active":"2024-05-09T05:01:39.865751+0000","last_became_peered":"2024-05-09T05:01:39.865751+0000","last_unstale":"2024-05-09T05:01:40.980784+0000","last_undegraded":"2024-05-09T05:01:40.980784+0000","last_fullsized":"2024-05-09T05:01:40.980784+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:30.919586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857772+0000","last_change":"2024-05-09T05:01:38.858711+0000","last_active":"2024-05-09T05:01:40.857772+0000","last_peered":"2024-05-09T05:01:40.857772+0000","last_clean":"2024-05-09T05:01:40.857772+0000","last_became_active":"2024-05-09T05:01:38.858438+0000","last_became_peered":"2024-05-09T05:01:38.858438+0000","last_unstale":"2024-05-09T05:01:40.857772+0000","last_undegraded":"2024-05-09T05:01:40.857772+0000","last_fullsized":"2024-05-09T05:01:40.857772+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:45.223268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857777+0000","last_change":"2024-05-09T05:01:39.881199+0000","last_active":"2024-05-09T05:01:40.857777+0000","last_peered":"2024-05-09T05:01:40.857777+0000","last_clean":"2024-05-09T05:01:40.857777+0000","last_became_active":"2024-05-09T05:01:39.880897+0000","last_became_peered":"2024-05-09T05:01:39.880897+0000","last_unstale":"2024-05-09T05:01:40.857777+0000","last_undegraded":"2024-05-09T05:01:40.857777+0000","last_fullsized":"2024-05-09T05:01:40.857777+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:03:16.895228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851249+0000","last_change":"2024-05-09T05:01:38.860738+0000","last_active":"2024-05-09T05:01:40.851249+0000","last_peered":"2024-05-09T05:01:40.851249+0000","last_clean":"2024-05-09T05:01:40.851249+0000","last_became_active":"2024-05-09T05:01:38.860472+0000","last_became_peered":"2024-05-09T05:01:38.860472+0000","last_unstale":"2024-05-09T05:01:40.851249+0000","last_undegraded":"2024-05-09T05:01:40.851249+0000","last_fullsized":"2024-05-09T05:01:40.851249+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:03.504627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358890+0000","last_change":"2024-05-09T05:01:39.874632+0000","last_active":"2024-05-09T05:01:41.358890+0000","last_peered":"2024-05-09T05:01:41.358890+0000","last_clean":"2024-05-09T05:01:41.358890+0000","last_became_active":"2024-05-09T05:01:39.871669+0000","last_became_peered":"2024-05-09T05:01:39.871669+0000","last_unstale":"2024-05-09T05:01:41.358890+0000","last_undegraded":"2024-05-09T05:01:41.358890+0000","last_fullsized":"2024-05-09T05:01:41.358890+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:22:24.375967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"50'3","reported_seq":17,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857601+0000","last_change":"2024-05-09T05:01:38.864487+0000","last_active":"2024-05-09T05:01:40.857601+0000","last_peered":"2024-05-09T05:01:40.857601+0000","last_clean":"2024-05-09T05:01:40.857601+0000","last_became_active":"2024-05-09T05:01:38.864194+0000","last_became_peered":"2024-05-09T05:01:38.864194+0000","last_unstale":"2024-05-09T05:01:40.857601+0000","last_undegraded":"2024-05-09T05:01:40.857601+0000","last_fullsized":"2024-05-09T05:01:40.857601+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:26:10.131662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857473+0000","last_change":"2024-05-09T05:01:39.862711+0000","last_active":"2024-05-09T05:01:40.857473+0000","last_peered":"2024-05-09T05:01:40.857473+0000","last_clean":"2024-05-09T05:01:40.857473+0000","last_became_active":"2024-05-09T05:01:39.862322+0000","last_became_peered":"2024-05-09T05:01:39.862322+0000","last_unstale":"2024-05-09T05:01:40.857473+0000","last_undegraded":"2024-05-09T05:01:40.857473+0000","last_fullsized":"2024-05-09T05:01:40.857473+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:48:03.359391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"50'6","reported_seq":20,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980752+0000","last_change":"2024-05-09T05:01:38.868251+0000","last_active":"2024-05-09T05:01:40.980752+0000","last_peered":"2024-05-09T05:01:40.980752+0000","last_clean":"2024-05-09T05:01:40.980752+0000","last_became_active":"2024-05-09T05:01:38.868026+0000","last_became_peered":"2024-05-09T05:01:38.868026+0000","last_unstale":"2024-05-09T05:01:40.980752+0000","last_undegraded":"2024-05-09T05:01:40.980752+0000","last_fullsized":"2024-05-09T05:01:40.980752+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:30:01.506489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.357854+0000","last_change":"2024-05-09T05:01:39.869503+0000","last_active":"2024-05-09T05:01:41.357854+0000","last_peered":"2024-05-09T05:01:41.357854+0000","last_clean":"2024-05-09T05:01:41.357854+0000","last_became_active":"2024-05-09T05:01:39.868873+0000","last_became_peered":"2024-05-09T05:01:39.868873+0000","last_unstale":"2024-05-09T05:01:41.357854+0000","last_undegraded":"2024-05-09T05:01:41.357854+0000","last_fullsized":"2024-05-09T05:01:41.357854+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:21.336536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851168+0000","last_change":"2024-05-09T05:01:38.859004+0000","last_active":"2024-05-09T05:01:40.851168+0000","last_peered":"2024-05-09T05:01:40.851168+0000","last_clean":"2024-05-09T05:01:40.851168+0000","last_became_active":"2024-05-09T05:01:38.858712+0000","last_became_peered":"2024-05-09T05:01:38.858712+0000","last_unstale":"2024-05-09T05:01:40.851168+0000","last_undegraded":"2024-05-09T05:01:40.851168+0000","last_fullsized":"2024-05-09T05:01:40.851168+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:58:53.883816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857379+0000","last_change":"2024-05-09T05:01:39.863597+0000","last_active":"2024-05-09T05:01:40.857379+0000","last_peered":"2024-05-09T05:01:40.857379+0000","last_clean":"2024-05-09T05:01:40.857379+0000","last_became_active":"2024-05-09T05:01:39.863310+0000","last_became_peered":"2024-05-09T05:01:39.863310+0000","last_unstale":"2024-05-09T05:01:40.857379+0000","last_undegraded":"2024-05-09T05:01:40.857379+0000","last_fullsized":"2024-05-09T05:01:40.857379+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:21.186391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.357824+0000","last_change":"2024-05-09T05:01:38.860319+0000","last_active":"2024-05-09T05:01:41.357824+0000","last_peered":"2024-05-09T05:01:41.357824+0000","last_clean":"2024-05-09T05:01:41.357824+0000","last_became_active":"2024-05-09T05:01:38.860033+0000","last_became_peered":"2024-05-09T05:01:38.860033+0000","last_unstale":"2024-05-09T05:01:41.357824+0000","last_undegraded":"2024-05-09T05:01:41.357824+0000","last_fullsized":"2024-05-09T05:01:41.357824+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:05.615229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980511+0000","last_change":"2024-05-09T05:01:39.860133+0000","last_active":"2024-05-09T05:01:40.980511+0000","last_peered":"2024-05-09T05:01:40.980511+0000","last_clean":"2024-05-09T05:01:40.980511+0000","last_became_active":"2024-05-09T05:01:39.859806+0000","last_became_peered":"2024-05-09T05:01:39.859806+0000","last_unstale":"2024-05-09T05:01:40.980511+0000","last_undegraded":"2024-05-09T05:01:40.980511+0000","last_fullsized":"2024-05-09T05:01:40.980511+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:35.258317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358372+0000","last_change":"2024-05-09T05:01:38.860319+0000","last_active":"2024-05-09T05:01:41.358372+0000","last_peered":"2024-05-09T05:01:41.358372+0000","last_clean":"2024-05-09T05:01:41.358372+0000","last_became_active":"2024-05-09T05:01:38.860025+0000","last_became_peered":"2024-05-09T05:01:38.860025+0000","last_unstale":"2024-05-09T05:01:41.358372+0000","last_undegraded":"2024-05-09T05:01:41.358372+0000","last_fullsized":"2024-05-09T05:01:41.358372+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:35:19.523791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857419+0000","last_change":"2024-05-09T05:01:39.872694+0000","last_active":"2024-05-09T05:01:40.857419+0000","last_peered":"2024-05-09T05:01:40.857419+0000","last_clean":"2024-05-09T05:01:40.857419+0000","last_became_active":"2024-05-09T05:01:39.872386+0000","last_became_peered":"2024-05-09T05:01:39.872386+0000","last_unstale":"2024-05-09T05:01:40.857419+0000","last_undegraded":"2024-05-09T05:01:40.857419+0000","last_fullsized":"2024-05-09T05:01:40.857419+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:26.273557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"50'8","reported_seq":22,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857335+0000","last_change":"2024-05-09T05:01:38.863353+0000","last_active":"2024-05-09T05:01:40.857335+0000","last_peered":"2024-05-09T05:01:40.857335+0000","last_clean":"2024-05-09T05:01:40.857335+0000","last_became_active":"2024-05-09T05:01:38.863084+0000","last_became_peered":"2024-05-09T05:01:38.863084+0000","last_unstale":"2024-05-09T05:01:40.857335+0000","last_undegraded":"2024-05-09T05:01:40.857335+0000","last_fullsized":"2024-05-09T05:01:40.857335+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:39:57.155429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857692+0000","last_change":"2024-05-09T05:01:39.868252+0000","last_active":"2024-05-09T05:01:40.857692+0000","last_peered":"2024-05-09T05:01:40.857692+0000","last_clean":"2024-05-09T05:01:40.857692+0000","last_became_active":"2024-05-09T05:01:39.867852+0000","last_became_peered":"2024-05-09T05:01:39.867852+0000","last_unstale":"2024-05-09T05:01:40.857692+0000","last_undegraded":"2024-05-09T05:01:40.857692+0000","last_fullsized":"2024-05-09T05:01:40.857692+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:37:47.075587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851773+0000","last_change":"2024-05-09T05:01:38.858925+0000","last_active":"2024-05-09T05:01:40.851773+0000","last_peered":"2024-05-09T05:01:40.851773+0000","last_clean":"2024-05-09T05:01:40.851773+0000","last_became_active":"2024-05-09T05:01:38.858625+0000","last_became_peered":"2024-05-09T05:01:38.858625+0000","last_unstale":"2024-05-09T05:01:40.851773+0000","last_undegraded":"2024-05-09T05:01:40.851773+0000","last_fullsized":"2024-05-09T05:01:40.851773+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:17:28.288161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358268+0000","last_change":"2024-05-09T05:01:39.880829+0000","last_active":"2024-05-09T05:01:41.358268+0000","last_peered":"2024-05-09T05:01:41.358268+0000","last_clean":"2024-05-09T05:01:41.358268+0000","last_became_active":"2024-05-09T05:01:39.880452+0000","last_became_peered":"2024-05-09T05:01:39.880452+0000","last_unstale":"2024-05-09T05:01:41.358268+0000","last_undegraded":"2024-05-09T05:01:41.358268+0000","last_fullsized":"2024-05-09T05:01:41.358268+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:21.186391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358473+0000","last_change":"2024-05-09T05:01:38.860324+0000","last_active":"2024-05-09T05:01:41.358473+0000","last_peered":"2024-05-09T05:01:41.358473+0000","last_clean":"2024-05-09T05:01:41.358473+0000","last_became_active":"2024-05-09T05:01:38.860006+0000","last_became_peered":"2024-05-09T05:01:38.860006+0000","last_unstale":"2024-05-09T05:01:41.358473+0000","last_undegraded":"2024-05-09T05:01:41.358473+0000","last_fullsized":"2024-05-09T05:01:41.358473+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:07:15.657753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851342+0000","last_change":"2024-05-09T05:01:39.860722+0000","last_active":"2024-05-09T05:01:40.851342+0000","last_peered":"2024-05-09T05:01:40.851342+0000","last_clean":"2024-05-09T05:01:40.851342+0000","last_became_active":"2024-05-09T05:01:39.858607+0000","last_became_peered":"2024-05-09T05:01:39.858607+0000","last_unstale":"2024-05-09T05:01:40.851342+0000","last_undegraded":"2024-05-09T05:01:40.851342+0000","last_fullsized":"2024-05-09T05:01:40.851342+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:48:49.629108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"50'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358453+0000","last_change":"2024-05-09T05:01:38.860226+0000","last_active":"2024-05-09T05:01:41.358453+0000","last_peered":"2024-05-09T05:01:41.358453+0000","last_clean":"2024-05-09T05:01:41.358453+0000","last_became_active":"2024-05-09T05:01:38.859683+0000","last_became_peered":"2024-05-09T05:01:38.859683+0000","last_unstale":"2024-05-09T05:01:41.358453+0000","last_undegraded":"2024-05-09T05:01:41.358453+0000","last_fullsized":"2024-05-09T05:01:41.358453+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:00:56.428799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857659+0000","last_change":"2024-05-09T05:01:39.862901+0000","last_active":"2024-05-09T05:01:40.857659+0000","last_peered":"2024-05-09T05:01:40.857659+0000","last_clean":"2024-05-09T05:01:40.857659+0000","last_became_active":"2024-05-09T05:01:39.862621+0000","last_became_peered":"2024-05-09T05:01:39.862621+0000","last_unstale":"2024-05-09T05:01:40.857659+0000","last_undegraded":"2024-05-09T05:01:40.857659+0000","last_fullsized":"2024-05-09T05:01:40.857659+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:17:38.783370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358545+0000","last_change":"2024-05-09T05:01:38.861565+0000","last_active":"2024-05-09T05:01:41.358545+0000","last_peered":"2024-05-09T05:01:41.358545+0000","last_clean":"2024-05-09T05:01:41.358545+0000","last_became_active":"2024-05-09T05:01:38.861245+0000","last_became_peered":"2024-05-09T05:01:38.861245+0000","last_unstale":"2024-05-09T05:01:41.358545+0000","last_undegraded":"2024-05-09T05:01:41.358545+0000","last_fullsized":"2024-05-09T05:01:41.358545+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:45.223268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358339+0000","last_change":"2024-05-09T05:01:39.869735+0000","last_active":"2024-05-09T05:01:41.358339+0000","last_peered":"2024-05-09T05:01:41.358339+0000","last_clean":"2024-05-09T05:01:41.358339+0000","last_became_active":"2024-05-09T05:01:39.869239+0000","last_became_peered":"2024-05-09T05:01:39.869239+0000","last_unstale":"2024-05-09T05:01:41.358339+0000","last_undegraded":"2024-05-09T05:01:41.358339+0000","last_fullsized":"2024-05-09T05:01:41.358339+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:53:26.273557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851280+0000","last_change":"2024-05-09T05:01:38.868383+0000","last_active":"2024-05-09T05:01:40.851280+0000","last_peered":"2024-05-09T05:01:40.851280+0000","last_clean":"2024-05-09T05:01:40.851280+0000","last_became_active":"2024-05-09T05:01:38.867935+0000","last_became_peered":"2024-05-09T05:01:38.867935+0000","last_unstale":"2024-05-09T05:01:40.851280+0000","last_undegraded":"2024-05-09T05:01:40.851280+0000","last_fullsized":"2024-05-09T05:01:40.851280+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:09:43.953892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358624+0000","last_change":"2024-05-09T05:01:39.868847+0000","last_active":"2024-05-09T05:01:41.358624+0000","last_peered":"2024-05-09T05:01:41.358624+0000","last_clean":"2024-05-09T05:01:41.358624+0000","last_became_active":"2024-05-09T05:01:39.868602+0000","last_became_peered":"2024-05-09T05:01:39.868602+0000","last_unstale":"2024-05-09T05:01:41.358624+0000","last_undegraded":"2024-05-09T05:01:41.358624+0000","last_fullsized":"2024-05-09T05:01:41.358624+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:57:48.604326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358419+0000","last_change":"2024-05-09T05:01:38.861303+0000","last_active":"2024-05-09T05:01:41.358419+0000","last_peered":"2024-05-09T05:01:41.358419+0000","last_clean":"2024-05-09T05:01:41.358419+0000","last_became_active":"2024-05-09T05:01:38.860616+0000","last_became_peered":"2024-05-09T05:01:38.860616+0000","last_unstale":"2024-05-09T05:01:41.358419+0000","last_undegraded":"2024-05-09T05:01:41.358419+0000","last_fullsized":"2024-05-09T05:01:41.358419+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:52.445863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358527+0000","last_change":"2024-05-09T05:01:39.875323+0000","last_active":"2024-05-09T05:01:41.358527+0000","last_peered":"2024-05-09T05:01:41.358527+0000","last_clean":"2024-05-09T05:01:41.358527+0000","last_became_active":"2024-05-09T05:01:39.875028+0000","last_became_peered":"2024-05-09T05:01:39.875028+0000","last_unstale":"2024-05-09T05:01:41.358527+0000","last_undegraded":"2024-05-09T05:01:41.358527+0000","last_fullsized":"2024-05-09T05:01:41.358527+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:06.942055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.357925+0000","last_change":"2024-05-09T05:01:38.866624+0000","last_active":"2024-05-09T05:01:41.357925+0000","last_peered":"2024-05-09T05:01:41.357925+0000","last_clean":"2024-05-09T05:01:41.357925+0000","last_became_active":"2024-05-09T05:01:38.866301+0000","last_became_peered":"2024-05-09T05:01:38.866301+0000","last_unstale":"2024-05-09T05:01:41.357925+0000","last_undegraded":"2024-05-09T05:01:41.357925+0000","last_fullsized":"2024-05-09T05:01:41.357925+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:26:10.131662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358294+0000","last_change":"2024-05-09T05:01:39.874994+0000","last_active":"2024-05-09T05:01:41.358294+0000","last_peered":"2024-05-09T05:01:41.358294+0000","last_clean":"2024-05-09T05:01:41.358294+0000","last_became_active":"2024-05-09T05:01:39.874713+0000","last_became_peered":"2024-05-09T05:01:39.874713+0000","last_unstale":"2024-05-09T05:01:41.358294+0000","last_undegraded":"2024-05-09T05:01:41.358294+0000","last_fullsized":"2024-05-09T05:01:41.358294+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:06.942055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"50'2","reported_seq":16,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358969+0000","last_change":"2024-05-09T05:01:38.867281+0000","last_active":"2024-05-09T05:01:41.358969+0000","last_peered":"2024-05-09T05:01:41.358969+0000","last_clean":"2024-05-09T05:01:41.358969+0000","last_became_active":"2024-05-09T05:01:38.867007+0000","last_became_peered":"2024-05-09T05:01:38.867007+0000","last_unstale":"2024-05-09T05:01:41.358969+0000","last_undegraded":"2024-05-09T05:01:41.358969+0000","last_fullsized":"2024-05-09T05:01:41.358969+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:28.819242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358407+0000","last_change":"2024-05-09T05:01:39.873250+0000","last_active":"2024-05-09T05:01:41.358407+0000","last_peered":"2024-05-09T05:01:41.358407+0000","last_clean":"2024-05-09T05:01:41.358407+0000","last_became_active":"2024-05-09T05:01:39.872934+0000","last_became_peered":"2024-05-09T05:01:39.872934+0000","last_unstale":"2024-05-09T05:01:41.358407+0000","last_undegraded":"2024-05-09T05:01:41.358407+0000","last_fullsized":"2024-05-09T05:01:41.358407+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:18:54.996982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"51'1","reported_seq":15,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.856206+0000","last_change":"2024-05-09T05:01:38.862288+0000","last_active":"2024-05-09T05:01:41.856206+0000","last_peered":"2024-05-09T05:01:41.856206+0000","last_clean":"2024-05-09T05:01:41.856206+0000","last_became_active":"2024-05-09T05:01:38.862002+0000","last_became_peered":"2024-05-09T05:01:38.862002+0000","last_unstale":"2024-05-09T05:01:41.856206+0000","last_undegraded":"2024-05-09T05:01:41.856206+0000","last_fullsized":"2024-05-09T05:01:41.856206+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:05:21.818769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358948+0000","last_change":"2024-05-09T05:01:39.878889+0000","last_active":"2024-05-09T05:01:41.358948+0000","last_peered":"2024-05-09T05:01:41.358948+0000","last_clean":"2024-05-09T05:01:41.358948+0000","last_became_active":"2024-05-09T05:01:39.874324+0000","last_became_peered":"2024-05-09T05:01:39.874324+0000","last_unstale":"2024-05-09T05:01:41.358948+0000","last_undegraded":"2024-05-09T05:01:41.358948+0000","last_fullsized":"2024-05-09T05:01:41.358948+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:39:58.160343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358466+0000","last_change":"2024-05-09T05:01:38.858855+0000","last_active":"2024-05-09T05:01:41.358466+0000","last_peered":"2024-05-09T05:01:41.358466+0000","last_clean":"2024-05-09T05:01:41.358466+0000","last_became_active":"2024-05-09T05:01:38.858570+0000","last_became_peered":"2024-05-09T05:01:38.858570+0000","last_unstale":"2024-05-09T05:01:41.358466+0000","last_undegraded":"2024-05-09T05:01:41.358466+0000","last_fullsized":"2024-05-09T05:01:41.358466+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:38.746187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358661+0000","last_change":"2024-05-09T05:01:39.870147+0000","last_active":"2024-05-09T05:01:41.358661+0000","last_peered":"2024-05-09T05:01:41.358661+0000","last_clean":"2024-05-09T05:01:41.358661+0000","last_became_active":"2024-05-09T05:01:39.869848+0000","last_became_peered":"2024-05-09T05:01:39.869848+0000","last_unstale":"2024-05-09T05:01:41.358661+0000","last_undegraded":"2024-05-09T05:01:41.358661+0000","last_fullsized":"2024-05-09T05:01:41.358661+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:48:03.359391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358583+0000","last_change":"2024-05-09T05:01:38.865398+0000","last_active":"2024-05-09T05:01:41.358583+0000","last_peered":"2024-05-09T05:01:41.358583+0000","last_clean":"2024-05-09T05:01:41.358583+0000","last_became_active":"2024-05-09T05:01:38.865098+0000","last_became_peered":"2024-05-09T05:01:38.865098+0000","last_unstale":"2024-05-09T05:01:41.358583+0000","last_undegraded":"2024-05-09T05:01:41.358583+0000","last_fullsized":"2024-05-09T05:01:41.358583+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:17:00.452604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851102+0000","last_change":"2024-05-09T05:01:39.863602+0000","last_active":"2024-05-09T05:01:40.851102+0000","last_peered":"2024-05-09T05:01:40.851102+0000","last_clean":"2024-05-09T05:01:40.851102+0000","last_became_active":"2024-05-09T05:01:39.863245+0000","last_became_peered":"2024-05-09T05:01:39.863245+0000","last_unstale":"2024-05-09T05:01:40.851102+0000","last_undegraded":"2024-05-09T05:01:40.851102+0000","last_fullsized":"2024-05-09T05:01:40.851102+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:33:27.012550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.358486+0000","last_change":"2024-05-09T05:01:38.863121+0000","last_active":"2024-05-09T05:01:41.358486+0000","last_peered":"2024-05-09T05:01:41.358486+0000","last_clean":"2024-05-09T05:01:41.358486+0000","last_became_active":"2024-05-09T05:01:38.862843+0000","last_became_peered":"2024-05-09T05:01:38.862843+0000","last_unstale":"2024-05-09T05:01:41.358486+0000","last_undegraded":"2024-05-09T05:01:41.358486+0000","last_fullsized":"2024-05-09T05:01:41.358486+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:05.615229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857557+0000","last_change":"2024-05-09T05:01:39.863381+0000","last_active":"2024-05-09T05:01:40.857557+0000","last_peered":"2024-05-09T05:01:40.857557+0000","last_clean":"2024-05-09T05:01:40.857557+0000","last_became_active":"2024-05-09T05:01:39.862839+0000","last_became_peered":"2024-05-09T05:01:39.862839+0000","last_unstale":"2024-05-09T05:01:40.857557+0000","last_undegraded":"2024-05-09T05:01:40.857557+0000","last_fullsized":"2024-05-09T05:01:40.857557+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:18:54.996982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.980497+0000","last_change":"2024-05-09T05:01:38.866869+0000","last_active":"2024-05-09T05:01:40.980497+0000","last_peered":"2024-05-09T05:01:40.980497+0000","last_clean":"2024-05-09T05:01:40.980497+0000","last_became_active":"2024-05-09T05:01:38.866569+0000","last_became_peered":"2024-05-09T05:01:38.866569+0000","last_unstale":"2024-05-09T05:01:40.980497+0000","last_undegraded":"2024-05-09T05:01:40.980497+0000","last_fullsized":"2024-05-09T05:01:40.980497+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:14:47.456838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.357865+0000","last_change":"2024-05-09T05:01:39.879631+0000","last_active":"2024-05-09T05:01:41.357865+0000","last_peered":"2024-05-09T05:01:41.357865+0000","last_clean":"2024-05-09T05:01:41.357865+0000","last_became_active":"2024-05-09T05:01:39.875087+0000","last_became_peered":"2024-05-09T05:01:39.875087+0000","last_unstale":"2024-05-09T05:01:41.357865+0000","last_undegraded":"2024-05-09T05:01:41.357865+0000","last_fullsized":"2024-05-09T05:01:41.357865+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:30.919586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.357900+0000","last_change":"2024-05-09T05:01:38.859275+0000","last_active":"2024-05-09T05:01:41.357900+0000","last_peered":"2024-05-09T05:01:41.357900+0000","last_clean":"2024-05-09T05:01:41.357900+0000","last_became_active":"2024-05-09T05:01:38.858976+0000","last_became_peered":"2024-05-09T05:01:38.858976+0000","last_unstale":"2024-05-09T05:01:41.357900+0000","last_undegraded":"2024-05-09T05:01:41.357900+0000","last_fullsized":"2024-05-09T05:01:41.357900+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:48:03.567166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.850893+0000","last_change":"2024-05-09T05:01:39.870972+0000","last_active":"2024-05-09T05:01:40.850893+0000","last_peered":"2024-05-09T05:01:40.850893+0000","last_clean":"2024-05-09T05:01:40.850893+0000","last_became_active":"2024-05-09T05:01:39.867594+0000","last_became_peered":"2024-05-09T05:01:39.867594+0000","last_unstale":"2024-05-09T05:01:40.850893+0000","last_undegraded":"2024-05-09T05:01:40.850893+0000","last_fullsized":"2024-05-09T05:01:40.850893+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:23:38.645452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851033+0000","last_change":"2024-05-09T05:01:38.865444+0000","last_active":"2024-05-09T05:01:40.851033+0000","last_peered":"2024-05-09T05:01:40.851033+0000","last_clean":"2024-05-09T05:01:40.851033+0000","last_became_active":"2024-05-09T05:01:38.865025+0000","last_became_peered":"2024-05-09T05:01:38.865025+0000","last_unstale":"2024-05-09T05:01:40.851033+0000","last_undegraded":"2024-05-09T05:01:40.851033+0000","last_fullsized":"2024-05-09T05:01:40.851033+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:39.777738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.851043+0000","last_change":"2024-05-09T05:01:39.867490+0000","last_active":"2024-05-09T05:01:40.851043+0000","last_peered":"2024-05-09T05:01:40.851043+0000","last_clean":"2024-05-09T05:01:40.851043+0000","last_became_active":"2024-05-09T05:01:39.867145+0000","last_became_peered":"2024-05-09T05:01:39.867145+0000","last_unstale":"2024-05-09T05:01:40.851043+0000","last_undegraded":"2024-05-09T05:01:40.851043+0000","last_fullsized":"2024-05-09T05:01:40.851043+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:38.840961+0000","last_clean_scrub_stamp":"2024-05-09T05:01:38.840961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:08:54.716934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:41.357792+0000","last_change":"2024-05-09T05:01:38.863680+0000","last_active":"2024-05-09T05:01:41.357792+0000","last_peered":"2024-05-09T05:01:41.357792+0000","last_clean":"2024-05-09T05:01:41.357792+0000","last_became_active":"2024-05-09T05:01:38.863378+0000","last_became_peered":"2024-05-09T05:01:38.863378+0000","last_unstale":"2024-05-09T05:01:41.357792+0000","last_undegraded":"2024-05-09T05:01:41.357792+0000","last_fullsized":"2024-05-09T05:01:41.357792+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:01:37.836047+0000","last_clean_scrub_stamp":"2024-05-09T05:01:37.836047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:06:28.931921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":167,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-05-09T05:01:40.857137+0000","last_change":"2024-05-09T05:00:08.667158+0000","last_active":"2024-05-09T05:01:40.857137+0000","last_peered":"2024-05-09T05:01:40.857137+0000","last_clean":"2024-05-09T05:01:40.857137+0000","last_became_active":"2024-05-09T05:00:08.666316+0000","last_became_peered":"2024-05-09T05:00:08.666316+0000","last_unstale":"2024-05-09T05:01:40.857137+0000","last_undegraded":"2024-05-09T05:01:40.857137+0000","last_fullsized":"2024-05-09T05:01:40.857137+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_clean_scrub_stamp":"2024-05-09T05:00:07.370119+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:20.234643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28204,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343636,"statfs":{"total":96636764160,"available":96607883264,"internally_reserved":0,"allocated":1212416,"data_stored":861719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":598016,"data_stored":264499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":265151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":622592,"data_stored":272803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31784,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340056,"statfs":{"total":96636764160,"available":96604217344,"internally_reserved":0,"allocated":618496,"data_stored":271247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":618496,"data_stored":271247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31780,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340060,"statfs":{"total":96636764160,"available":96604221440,"internally_reserved":0,"allocated":614400,"data_stored":271281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1220608,"data_stored":863753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:01:44.279 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-09T05:01:44.279 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-09T05:01:44.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.426+0000 7fbc36903640 1 Processor -- start 2024-05-09T05:01:44.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.426+0000 7fbc36903640 1 -- start start 2024-05-09T05:01:44.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.427+0000 7fbc36903640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbc30146930 0x7fbc300d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.427+0000 7fbc36903640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc300d09e0 0x7fbc30073580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.427+0000 7fbc36903640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc301609e0 con 0x7fbc30160600 2024-05-09T05:01:44.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.427+0000 7fbc36903640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fbc300730f0 con 0x7fbc30146930 2024-05-09T05:01:44.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.427+0000 7fbc36903640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbc30071fb0 con 0x7fbc300d09e0 2024-05-09T05:01:44.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.427+0000 7fbc2f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbc30146930 0x7fbc300d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.427+0000 7fbc2f7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbc30146930 0x7fbc300d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37746/0 (socket says 172.21.15.60:37746) 2024-05-09T05:01:44.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.427+0000 7fbc2f7fe640 1 -- 172.21.15.60:0/1953723376 learned_addr learned my addr 172.21.15.60:0/1953723376 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:44.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.427+0000 7fbc34ea0640 1 --2- 172.21.15.60:0/1953723376 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc300d09e0 0x7fbc30073580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.427+0000 7fbc2f7fe640 1 -- 172.21.15.60:0/1953723376 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc300d09e0 msgr2=0x7fbc30073580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:44.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.427+0000 7fbc2f7fe640 1 --2- 172.21.15.60:0/1953723376 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc300d09e0 0x7fbc30073580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.427+0000 7fbc2f7fe640 1 -- 172.21.15.60:0/1953723376 >> v1:172.21.15.114:6789/0 conn(0x7fbc30160600 legacy=0x7fbc300718a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:44.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.427+0000 7fbc2f7fe640 1 -- 172.21.15.60:0/1953723376 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc30072280 con 0x7fbc30146930 2024-05-09T05:01:44.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.427+0000 7fbc34ea0640 1 --2- 172.21.15.60:0/1953723376 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc300d09e0 0x7fbc30073580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:44.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.428+0000 7fbc2f7fe640 1 --2- 172.21.15.60:0/1953723376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc30146930 0x7fbc300d04a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbc30073270 tx=0x7fbc2402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=481a5a1053b5892e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:44.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.428+0000 7fbc2effd640 1 -- 172.21.15.60:0/1953723376 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc24037070 con 0x7fbc30146930 2024-05-09T05:01:44.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.428+0000 7fbc2effd640 1 -- 172.21.15.60:0/1953723376 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc24032440 con 0x7fbc30146930 2024-05-09T05:01:44.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.428+0000 7fbc2effd640 1 -- 172.21.15.60:0/1953723376 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc24032810 con 0x7fbc30146930 2024-05-09T05:01:44.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.429+0000 7fbc36903640 1 -- 172.21.15.60:0/1953723376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc30146930 msgr2=0x7fbc300d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:44.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.429+0000 7fbc36903640 1 --2- 172.21.15.60:0/1953723376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc30146930 0x7fbc300d04a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbc30073270 tx=0x7fbc2402a250 comp rx=0 tx=0).stop 2024-05-09T05:01:44.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.429+0000 7fbc36903640 1 -- 172.21.15.60:0/1953723376 shutdown_connections 2024-05-09T05:01:44.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.429+0000 7fbc36903640 1 --2- 172.21.15.60:0/1953723376 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc300d09e0 0x7fbc30073580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.429+0000 7fbc36903640 1 --2- 172.21.15.60:0/1953723376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc30146930 0x7fbc300d04a0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.429+0000 7fbc36903640 1 -- 172.21.15.60:0/1953723376 >> 172.21.15.60:0/1953723376 conn(0x7fbc30070000 msgr2=0x7fbc3006c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:44.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.429+0000 7fbc36903640 1 -- 172.21.15.60:0/1953723376 shutdown_connections 2024-05-09T05:01:44.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.429+0000 7fbc36903640 1 -- 172.21.15.60:0/1953723376 wait complete. 2024-05-09T05:01:44.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.430+0000 7fbc36903640 1 Processor -- start 2024-05-09T05:01:44.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.430+0000 7fbc36903640 1 -- start start 2024-05-09T05:01:44.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.431+0000 7fbc36903640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc300d09e0 0x7fbc300ce290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.431+0000 7fbc36903640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc30146930 0x7fbc300c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.431+0000 7fbc36903640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc30160600 0x7fbc300c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.431+0000 7fbc36903640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbc30072840 con 0x7fbc30146930 2024-05-09T05:01:44.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.431+0000 7fbc36903640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbc30072d80 con 0x7fbc30160600 2024-05-09T05:01:44.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.431+0000 7fbc36903640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbc30073ac0 con 0x7fbc300d09e0 2024-05-09T05:01:44.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.431+0000 7fbc2f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc30146930 0x7fbc300c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.431+0000 7fbc2f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc30146930 0x7fbc300c8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37752/0 (socket says 172.21.15.60:37752) 2024-05-09T05:01:44.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.431+0000 7fbc2f7fe640 1 -- 172.21.15.60:0/2082470937 learned_addr learned my addr 172.21.15.60:0/2082470937 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:44.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.431+0000 7fbc2ffff640 1 --2- 172.21.15.60:0/2082470937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc300d09e0 0x7fbc300ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.431+0000 7fbc34ea0640 1 --2- 172.21.15.60:0/2082470937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc30160600 0x7fbc300c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.431+0000 7fbc2f7fe640 1 -- 172.21.15.60:0/2082470937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc300d09e0 msgr2=0x7fbc300ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:44.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.431+0000 7fbc2f7fe640 1 --2- 172.21.15.60:0/2082470937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc300d09e0 0x7fbc300ce290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.431+0000 7fbc2f7fe640 1 -- 172.21.15.60:0/2082470937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc30160600 msgr2=0x7fbc300c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:44.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.431+0000 7fbc2f7fe640 1 --2- 172.21.15.60:0/2082470937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc30160600 0x7fbc300c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.431+0000 7fbc2f7fe640 1 -- 172.21.15.60:0/2082470937 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc300ced50 con 0x7fbc30146930 2024-05-09T05:01:44.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.431+0000 7fbc34ea0640 1 --2- 172.21.15.60:0/2082470937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc30160600 0x7fbc300c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:44.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.432+0000 7fbc2f7fe640 1 --2- 172.21.15.60:0/2082470937 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc30146930 0x7fbc300c8de0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbc24000c00 tx=0x7fbc240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:44.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.432+0000 7fbc2cff9640 1 -- 172.21.15.60:0/2082470937 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc24041020 con 0x7fbc30146930 2024-05-09T05:01:44.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.432+0000 7fbc2cff9640 1 -- 172.21.15.60:0/2082470937 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc240306c0 con 0x7fbc30146930 2024-05-09T05:01:44.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.432+0000 7fbc2cff9640 1 -- 172.21.15.60:0/2082470937 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc24037040 con 0x7fbc30146930 2024-05-09T05:01:44.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.432+0000 7fbc36903640 1 -- 172.21.15.60:0/2082470937 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc300cf030 con 0x7fbc30146930 2024-05-09T05:01:44.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.432+0000 7fbc36903640 1 -- 172.21.15.60:0/2082470937 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc300cf5f0 con 0x7fbc30146930 2024-05-09T05:01:44.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.433+0000 7fbc36903640 1 -- 172.21.15.60:0/2082470937 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc30051040 con 0x7fbc30146930 2024-05-09T05:01:44.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.433+0000 7fbc2cff9640 1 -- 172.21.15.60:0/2082470937 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbc24044020 con 0x7fbc30146930 2024-05-09T05:01:44.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.434+0000 7fbc2cff9640 1 --2- 172.21.15.60:0/2082470937 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbc1c0762d0 0x7fbc1c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.434+0000 7fbc2cff9640 1 -- 172.21.15.60:0/2082470937 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fbc240b7e80 con 0x7fbc30146930 2024-05-09T05:01:44.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.434+0000 7fbc2ffff640 1 --2- 172.21.15.60:0/2082470937 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbc1c0762d0 0x7fbc1c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.435+0000 7fbc2ffff640 1 --2- 172.21.15.60:0/2082470937 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbc1c0762d0 0x7fbc1c078790 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbc300d0310 tx=0x7fbc20009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:44.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.438+0000 7fbc2cff9640 1 -- 172.21.15.60:0/2082470937 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbc2404b030 con 0x7fbc30146930 2024-05-09T05:01:44.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.702+0000 7fbc36903640 1 -- 172.21.15.60:0/2082470937 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fbc30071ab0 con 0x7fbc30146930 2024-05-09T05:01:44.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.704+0000 7fbc2cff9640 1 -- 172.21.15.60:0/2082470937 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) ==== 190+0+147 (secure 0 0 0) 0x7fbc24081640 con 0x7fbc30146930 2024-05-09T05:01:44.704 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2024-05-09T05:01:44.705 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQAtWDxmiEa1CBAAm5ehYW6MVg0B6WUq8TxNbg== 2024-05-09T05:01:44.705 INFO:teuthology.orchestra.run.smithi060.stdout: caps mds = "allow" 2024-05-09T05:01:44.705 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "allow r" 2024-05-09T05:01:44.705 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-05-09T05:01:44.705 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2024-05-09T05:01:44.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.707+0000 7fbc36903640 1 -- 172.21.15.60:0/2082470937 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbc1c0762d0 msgr2=0x7fbc1c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:44.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.707+0000 7fbc36903640 1 --2- 172.21.15.60:0/2082470937 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbc1c0762d0 0x7fbc1c078790 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbc300d0310 tx=0x7fbc20009040 comp rx=0 tx=0).stop 2024-05-09T05:01:44.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.707+0000 7fbc36903640 1 -- 172.21.15.60:0/2082470937 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc30146930 msgr2=0x7fbc300c8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:44.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.707+0000 7fbc36903640 1 --2- 172.21.15.60:0/2082470937 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc30146930 0x7fbc300c8de0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbc24000c00 tx=0x7fbc240046c0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.707+0000 7fbc36903640 1 -- 172.21.15.60:0/2082470937 shutdown_connections 2024-05-09T05:01:44.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.707+0000 7fbc36903640 1 --2- 172.21.15.60:0/2082470937 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbc1c0762d0 0x7fbc1c078790 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.707+0000 7fbc36903640 1 --2- 172.21.15.60:0/2082470937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc30160600 0x7fbc300c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.707+0000 7fbc36903640 1 --2- 172.21.15.60:0/2082470937 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc30146930 0x7fbc300c8de0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.707+0000 7fbc36903640 1 --2- 172.21.15.60:0/2082470937 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbc300d09e0 0x7fbc300ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.707+0000 7fbc36903640 1 -- 172.21.15.60:0/2082470937 >> 172.21.15.60:0/2082470937 conn(0x7fbc30070000 msgr2=0x7fbc300b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:44.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.707+0000 7fbc36903640 1 -- 172.21.15.60:0/2082470937 shutdown_connections 2024-05-09T05:01:44.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.707+0000 7fbc36903640 1 -- 172.21.15.60:0/2082470937 wait complete. 2024-05-09T05:01:44.719 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:44.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.822+0000 7f8894b2c640 1 Processor -- start 2024-05-09T05:01:44.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.822+0000 7f8894b2c640 1 -- start start 2024-05-09T05:01:44.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.823+0000 7f8894b2c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f889015f300 0x7f88901639f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.823+0000 7f8894b2c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f889006ff60 con 0x7f889006f9f0 2024-05-09T05:01:44.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.823+0000 7f8894b2c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8890070240 con 0x7f88901606f0 2024-05-09T05:01:44.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.823+0000 7f8894b2c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8890070520 con 0x7f889015f300 2024-05-09T05:01:44.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.823+0000 7f888dd9b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f889006f9f0 0x7f88900d33f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57770/0 (socket says 172.21.15.60:57770) 2024-05-09T05:01:44.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.823+0000 7f888dd9b640 1 -- 172.21.15.60:0/3146995820 learned_addr learned my addr 172.21.15.60:0/3146995820 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:44.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.824+0000 7f888e59c640 1 --2- 172.21.15.60:0/3146995820 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f889015f300 0x7f88901639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.824+0000 7f888d59a640 1 -- 172.21.15.60:0/3146995820 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1638317488 0 0) 0x7f889006ff60 con 0x7f889006f9f0 2024-05-09T05:01:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.824+0000 7f888d59a640 1 -- 172.21.15.60:0/3146995820 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f887c003520 con 0x7f889006f9f0 2024-05-09T05:01:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.824+0000 7f888d59a640 1 -- 172.21.15.60:0/3146995820 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3168593923 0 0) 0x7f887c003520 con 0x7f889006f9f0 2024-05-09T05:01:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.824+0000 7f888d59a640 1 -- 172.21.15.60:0/3146995820 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f889015f300 msgr2=0x7f88901639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.824+0000 7f888d59a640 1 --2- 172.21.15.60:0/3146995820 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f889015f300 0x7f88901639f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.824+0000 7f888d59a640 1 -- 172.21.15.60:0/3146995820 >> v1:172.21.15.114:6789/0 conn(0x7f88901606f0 legacy=0x7f889006ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.824+0000 7f888d59a640 1 -- 172.21.15.60:0/3146995820 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889006e740 con 0x7f889006f9f0 2024-05-09T05:01:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.824+0000 7f888d59a640 1 -- 172.21.15.60:0/3146995820 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f88780016f0 con 0x7f889006f9f0 2024-05-09T05:01:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.824+0000 7f888d59a640 1 -- 172.21.15.60:0/3146995820 >> v1:172.21.15.60:6789/0 conn(0x7f889006f9f0 legacy=0x7f88900d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.824+0000 7f888d59a640 1 --2- 172.21.15.60:0/3146995820 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f889015f300 0x7f887c003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.824+0000 7f888d59a640 1 --2- 172.21.15.60:0/3146995820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c004000 0x7f887c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.824+0000 7f888d59a640 1 --2- 172.21.15.60:0/3146995820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f887c004d40 0x7f887c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.824+0000 7f888d59a640 1 -- 172.21.15.60:0/3146995820 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8890070520 con 0x7f887c004d40 2024-05-09T05:01:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.824+0000 7f888d59a640 1 -- 172.21.15.60:0/3146995820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f887c00bb50 con 0x7f887c004000 2024-05-09T05:01:44.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.824+0000 7f888d59a640 1 -- 172.21.15.60:0/3146995820 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f887c00bd50 con 0x7f889015f300 2024-05-09T05:01:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.824+0000 7f888ed9d640 1 --2- 172.21.15.60:0/3146995820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f887c004d40 0x7f887c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.824+0000 7f888e59c640 1 --2- 172.21.15.60:0/3146995820 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f889015f300 0x7f887c003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.824+0000 7f888dd9b640 1 --2- 172.21.15.60:0/3146995820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c004000 0x7f887c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.825+0000 7f888ed9d640 1 -- 172.21.15.60:0/3146995820 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f889015f300 msgr2=0x7f887c003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.825+0000 7f888ed9d640 1 --2- 172.21.15.60:0/3146995820 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f889015f300 0x7f887c003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.825+0000 7f888ed9d640 1 -- 172.21.15.60:0/3146995820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c004000 msgr2=0x7f887c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.825+0000 7f888ed9d640 1 --2- 172.21.15.60:0/3146995820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c004000 0x7f887c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.825+0000 7f888ed9d640 1 -- 172.21.15.60:0/3146995820 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f887c003520 con 0x7f887c004d40 2024-05-09T05:01:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.825+0000 7f888ed9d640 1 --2- 172.21.15.60:0/3146995820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f887c004d40 0x7f887c0051c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f8888001790 tx=0x7f887c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac399e034486771 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.826+0000 7f888d59a640 1 -- 172.21.15.60:0/3146995820 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88880013b0 con 0x7f887c004d40 2024-05-09T05:01:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.826+0000 7f888d59a640 1 -- 172.21.15.60:0/3146995820 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88880052d0 con 0x7f887c004d40 2024-05-09T05:01:44.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.826+0000 7f888d59a640 1 -- 172.21.15.60:0/3146995820 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888802aae0 con 0x7f887c004d40 2024-05-09T05:01:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.826+0000 7f8894b2c640 1 -- 172.21.15.60:0/3146995820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f887c004d40 msgr2=0x7f887c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.826+0000 7f8894b2c640 1 --2- 172.21.15.60:0/3146995820 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f887c004d40 0x7f887c0051c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f8888001790 tx=0x7f887c005830 comp rx=0 tx=0).stop 2024-05-09T05:01:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.826+0000 7f888ed9d640 1 -- 172.21.15.60:0/3146995820 reap_dead start 2024-05-09T05:01:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.826+0000 7f8894b2c640 1 -- 172.21.15.60:0/3146995820 shutdown_connections 2024-05-09T05:01:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.826+0000 7f8894b2c640 1 -- 172.21.15.60:0/3146995820 >> 172.21.15.60:0/3146995820 conn(0x7f889009b850 msgr2=0x7f889009bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.826+0000 7f8894b2c640 1 -- 172.21.15.60:0/3146995820 shutdown_connections 2024-05-09T05:01:44.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.826+0000 7f8894b2c640 1 -- 172.21.15.60:0/3146995820 wait complete. 2024-05-09T05:01:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.828+0000 7f8894b2c640 1 Processor -- start 2024-05-09T05:01:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.828+0000 7f8894b2c640 1 -- start start 2024-05-09T05:01:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.828+0000 7f8894b2c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c004d40 0x7f889009ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.828+0000 7f8894b2c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f889009e8d0 0x7f8890158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.828+0000 7f8894b2c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88900af4d0 0x7f88900af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.828+0000 7f8894b2c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f88900d3c80 con 0x7f889009e8d0 2024-05-09T05:01:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.828+0000 7f8894b2c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f88900d3b00 con 0x7f887c004d40 2024-05-09T05:01:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.828+0000 7f8894b2c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f889006f650 con 0x7f88900af4d0 2024-05-09T05:01:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.828+0000 7f888dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f889009e8d0 0x7f8890158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.829+0000 7f888dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f889009e8d0 0x7f8890158150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37758/0 (socket says 172.21.15.60:37758) 2024-05-09T05:01:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.829+0000 7f888dd9b640 1 -- 172.21.15.60:0/2112889541 learned_addr learned my addr 172.21.15.60:0/2112889541 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.829+0000 7f888e59c640 1 --2- 172.21.15.60:0/2112889541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c004d40 0x7f889009ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.829+0000 7f888ed9d640 1 --2- 172.21.15.60:0/2112889541 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88900af4d0 0x7f88900af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.829+0000 7f888dd9b640 1 -- 172.21.15.60:0/2112889541 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88900af4d0 msgr2=0x7f88900af950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.829+0000 7f888dd9b640 1 --2- 172.21.15.60:0/2112889541 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88900af4d0 0x7f88900af950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.829+0000 7f888dd9b640 1 -- 172.21.15.60:0/2112889541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c004d40 msgr2=0x7f889009ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.829+0000 7f888dd9b640 1 --2- 172.21.15.60:0/2112889541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c004d40 0x7f889009ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.829+0000 7f888dd9b640 1 -- 172.21.15.60:0/2112889541 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889009c620 con 0x7f889009e8d0 2024-05-09T05:01:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.829+0000 7f888dd9b640 1 --2- 172.21.15.60:0/2112889541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f889009e8d0 0x7f8890158150 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f8878007310 tx=0x7f88780077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.829+0000 7f8886ffd640 1 -- 172.21.15.60:0/2112889541 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887801a070 con 0x7f889009e8d0 2024-05-09T05:01:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.829+0000 7f8886ffd640 1 -- 172.21.15.60:0/2112889541 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f887800f940 con 0x7f889009e8d0 2024-05-09T05:01:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.829+0000 7f8894b2c640 1 -- 172.21.15.60:0/2112889541 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f889009c890 con 0x7f889009e8d0 2024-05-09T05:01:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.829+0000 7f8886ffd640 1 -- 172.21.15.60:0/2112889541 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887800fd30 con 0x7f889009e8d0 2024-05-09T05:01:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.829+0000 7f8894b2c640 1 -- 172.21.15.60:0/2112889541 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8890158690 con 0x7f889009e8d0 2024-05-09T05:01:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.830+0000 7f8894b2c640 1 -- 172.21.15.60:0/2112889541 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88900d1520 con 0x7f889009e8d0 2024-05-09T05:01:44.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.831+0000 7f8886ffd640 1 -- 172.21.15.60:0/2112889541 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f887801d070 con 0x7f889009e8d0 2024-05-09T05:01:44.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.831+0000 7f8886ffd640 1 --2- 172.21.15.60:0/2112889541 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f88700762d0 0x7f8870078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:44.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.831+0000 7f8886ffd640 1 -- 172.21.15.60:0/2112889541 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f88780970b0 con 0x7f889009e8d0 2024-05-09T05:01:44.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.832+0000 7f888e59c640 1 --2- 172.21.15.60:0/2112889541 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f88700762d0 0x7f8870078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:44.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.832+0000 7f888e59c640 1 --2- 172.21.15.60:0/2112889541 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f88700762d0 0x7f8870078790 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8880014510 tx=0x7f888000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:44.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:44.836+0000 7f8886ffd640 1 -- 172.21.15.60:0/2112889541 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f88900d1520 con 0x7f889009e8d0 2024-05-09T05:01:45.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.064+0000 7f8894b2c640 1 -- 172.21.15.60:0/2112889541 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8890051040 con 0x7f889009e8d0 2024-05-09T05:01:45.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.065+0000 7f8886ffd640 1 -- 172.21.15.60:0/2112889541 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5187 (secure 0 0 0) 0x7f8878060230 con 0x7f889009e8d0 2024-05-09T05:01:45.065 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:45.066 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3604145570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3604145570},{"type":"v1","addr":"172.21.15.60:6834","nonce":3604145570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3949194455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3949194455},{"type":"v1","addr":"172.21.15.114:6836","nonce":3949194455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3288630382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3288630382},{"type":"v1","addr":"172.21.15.114:6837","nonce":3288630382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:39.237498+0000","modified":"2024-05-09T05:01:42.253278+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5415":{"gid":5415,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2425263408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2425263408},{"type":"v1","addr":"172.21.15.60:6835","nonce":2425263408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5415,"qdb_cluster":[5415]},"id":3}]} 2024-05-09T05:01:45.066 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 27 2024-05-09T05:01:45.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.067+0000 7f8894b2c640 1 -- 172.21.15.60:0/2112889541 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f88700762d0 msgr2=0x7f8870078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:45.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.067+0000 7f8894b2c640 1 --2- 172.21.15.60:0/2112889541 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f88700762d0 0x7f8870078790 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8880014510 tx=0x7f888000b040 comp rx=0 tx=0).stop 2024-05-09T05:01:45.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.067+0000 7f8894b2c640 1 -- 172.21.15.60:0/2112889541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f889009e8d0 msgr2=0x7f8890158150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:45.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.067+0000 7f8894b2c640 1 --2- 172.21.15.60:0/2112889541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f889009e8d0 0x7f8890158150 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f8878007310 tx=0x7f88780077d0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.067+0000 7f8894b2c640 1 -- 172.21.15.60:0/2112889541 shutdown_connections 2024-05-09T05:01:45.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.067+0000 7f8894b2c640 1 --2- 172.21.15.60:0/2112889541 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f88700762d0 0x7f8870078790 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.067+0000 7f8894b2c640 1 --2- 172.21.15.60:0/2112889541 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f88900af4d0 0x7f88900af950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.067+0000 7f8894b2c640 1 --2- 172.21.15.60:0/2112889541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f889009e8d0 0x7f8890158150 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.067+0000 7f8894b2c640 1 --2- 172.21.15.60:0/2112889541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c004d40 0x7f889009ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.067+0000 7f8894b2c640 1 -- 172.21.15.60:0/2112889541 >> 172.21.15.60:0/2112889541 conn(0x7f889009b850 msgr2=0x7f8890099e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:45.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.067+0000 7f8894b2c640 1 -- 172.21.15.60:0/2112889541 shutdown_connections 2024-05-09T05:01:45.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.067+0000 7f8894b2c640 1 -- 172.21.15.60:0/2112889541 wait complete. 2024-05-09T05:01:45.481 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T05:01:39.237498+0000', 'modified': '2024-05-09T05:01:42.253278+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5415}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5415': {'gid': 5415, 'name': 'c', 'rank': 0, 'incarnation': 23, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6835/2425263408', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6833', 'nonce': 2425263408}, {'type': 'v1', 'addr': '172.21.15.60:6835', 'nonce': 2425263408}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5415, 'qdb_cluster': [5415]} 2024-05-09T05:01:45.481 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T05:01:45.481 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T05:01:45.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.592+0000 7fa9e60d3640 1 Processor -- start 2024-05-09T05:01:45.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.592+0000 7fa9e60d3640 1 -- start start 2024-05-09T05:01:45.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.593+0000 7fa9e60d3640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa9e015f300 0x7fa9e01639f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.593+0000 7fa9e60d3640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa9e006f9f0 0x7fa9e00d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.593+0000 7fa9e60d3640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9e006ff60 con 0x7fa9e00d3930 2024-05-09T05:01:45.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.593+0000 7fa9e60d3640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fa9e0070240 con 0x7fa9e006f9f0 2024-05-09T05:01:45.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.593+0000 7fa9e60d3640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fa9e0070520 con 0x7fa9e015f300 2024-05-09T05:01:45.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.593+0000 7fa9d7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa9e006f9f0 0x7fa9e00d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.593+0000 7fa9d7fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa9e006f9f0 0x7fa9e00d33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37772/0 (socket says 172.21.15.60:37772) 2024-05-09T05:01:45.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.593+0000 7fa9d7fff640 1 -- 172.21.15.60:0/4096139085 learned_addr learned my addr 172.21.15.60:0/4096139085 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:45.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.593+0000 7fa9df7fe640 1 --2- 172.21.15.60:0/4096139085 >> v2:172.21.15.114:3300/0 conn(0x7fa9e015f300 0x7fa9e01639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.593+0000 7fa9d7fff640 1 -- 172.21.15.60:0/4096139085 >> v2:172.21.15.114:3300/0 conn(0x7fa9e015f300 msgr2=0x7fa9e01639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:45.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.593+0000 7fa9d7fff640 1 --2- 172.21.15.60:0/4096139085 >> v2:172.21.15.114:3300/0 conn(0x7fa9e015f300 0x7fa9e01639f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.593+0000 7fa9d7fff640 1 -- 172.21.15.60:0/4096139085 >> v1:172.21.15.114:6789/0 conn(0x7fa9e00d3930 legacy=0x7fa9e006ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:45.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.593+0000 7fa9d7fff640 1 -- 172.21.15.60:0/4096139085 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e006e740 con 0x7fa9e006f9f0 2024-05-09T05:01:45.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.594+0000 7fa9d7fff640 1 --2- 172.21.15.60:0/4096139085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e006f9f0 0x7fa9e00d33f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa9e00d4fb0 tx=0x7fa9c802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa64d028d7d56de server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:45.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.594+0000 7fa9deffd640 1 -- 172.21.15.60:0/4096139085 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8037070 con 0x7fa9e006f9f0 2024-05-09T05:01:45.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.594+0000 7fa9deffd640 1 -- 172.21.15.60:0/4096139085 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9c8032440 con 0x7fa9e006f9f0 2024-05-09T05:01:45.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.594+0000 7fa9deffd640 1 -- 172.21.15.60:0/4096139085 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8032810 con 0x7fa9e006f9f0 2024-05-09T05:01:45.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.595+0000 7fa9e60d3640 1 -- 172.21.15.60:0/4096139085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e006f9f0 msgr2=0x7fa9e00d33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:45.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.595+0000 7fa9e60d3640 1 --2- 172.21.15.60:0/4096139085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e006f9f0 0x7fa9e00d33f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa9e00d4fb0 tx=0x7fa9c802a270 comp rx=0 tx=0).stop 2024-05-09T05:01:45.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.595+0000 7fa9e60d3640 1 -- 172.21.15.60:0/4096139085 shutdown_connections 2024-05-09T05:01:45.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.595+0000 7fa9e60d3640 1 --2- 172.21.15.60:0/4096139085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e006f9f0 0x7fa9e00d33f0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.595+0000 7fa9e60d3640 1 --2- 172.21.15.60:0/4096139085 >> v2:172.21.15.114:3300/0 conn(0x7fa9e015f300 0x7fa9e01639f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.595+0000 7fa9e60d3640 1 -- 172.21.15.60:0/4096139085 >> 172.21.15.60:0/4096139085 conn(0x7fa9e009b850 msgr2=0x7fa9e009bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:45.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.595+0000 7fa9e60d3640 1 -- 172.21.15.60:0/4096139085 shutdown_connections 2024-05-09T05:01:45.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.595+0000 7fa9e60d3640 1 -- 172.21.15.60:0/4096139085 wait complete. 2024-05-09T05:01:45.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.596+0000 7fa9e60d3640 1 Processor -- start 2024-05-09T05:01:45.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.596+0000 7fa9e60d3640 1 -- start start 2024-05-09T05:01:45.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.597+0000 7fa9e60d3640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e006f9f0 0x7fa9e00ac440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.597+0000 7fa9e60d3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e00d3930 0x7fa9e00aed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.597+0000 7fa9e60d3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e015f300 0x7fa9e015c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.597+0000 7fa9e60d3640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e006f650 con 0x7fa9e015f300 2024-05-09T05:01:45.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.597+0000 7fa9e60d3640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e0163f30 con 0x7fa9e00d3930 2024-05-09T05:01:45.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.597+0000 7fa9e60d3640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa9e006f7d0 con 0x7fa9e006f9f0 2024-05-09T05:01:45.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.597+0000 7fa9dffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e015f300 0x7fa9e015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.597+0000 7fa9dffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e015f300 0x7fa9e015c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37788/0 (socket says 172.21.15.60:37788) 2024-05-09T05:01:45.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.597+0000 7fa9dffff640 1 -- 172.21.15.60:0/2663720785 learned_addr learned my addr 172.21.15.60:0/2663720785 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:45.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.597+0000 7fa9df7fe640 1 --2- 172.21.15.60:0/2663720785 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e006f9f0 0x7fa9e00ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.597+0000 7fa9d7fff640 1 --2- 172.21.15.60:0/2663720785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e00d3930 0x7fa9e00aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.597+0000 7fa9dffff640 1 -- 172.21.15.60:0/2663720785 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e006f9f0 msgr2=0x7fa9e00ac440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:45.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.597+0000 7fa9dffff640 1 --2- 172.21.15.60:0/2663720785 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e006f9f0 0x7fa9e00ac440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.597+0000 7fa9dffff640 1 -- 172.21.15.60:0/2663720785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e00d3930 msgr2=0x7fa9e00aed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:45.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.597+0000 7fa9dffff640 1 --2- 172.21.15.60:0/2663720785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e00d3930 0x7fa9e00aed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.597+0000 7fa9dffff640 1 -- 172.21.15.60:0/2663720785 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e00b0ba0 con 0x7fa9e015f300 2024-05-09T05:01:45.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.597+0000 7fa9dffff640 1 --2- 172.21.15.60:0/2663720785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e015f300 0x7fa9e015c280 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa9d800c7a0 tx=0x7fa9d800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:45.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.598+0000 7fa9dcff9640 1 -- 172.21.15.60:0/2663720785 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8003b70 con 0x7fa9e015f300 2024-05-09T05:01:45.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.598+0000 7fa9dcff9640 1 -- 172.21.15.60:0/2663720785 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9d8003d10 con 0x7fa9e015f300 2024-05-09T05:01:45.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.598+0000 7fa9dcff9640 1 -- 172.21.15.60:0/2663720785 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d800aa70 con 0x7fa9e015f300 2024-05-09T05:01:45.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.598+0000 7fa9e60d3640 1 -- 172.21.15.60:0/2663720785 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e00b0e80 con 0x7fa9e015f300 2024-05-09T05:01:45.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.598+0000 7fa9e60d3640 1 -- 172.21.15.60:0/2663720785 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9e00b1720 con 0x7fa9e015f300 2024-05-09T05:01:45.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.599+0000 7fa9dcff9640 1 -- 172.21.15.60:0/2663720785 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa9d800ac10 con 0x7fa9e015f300 2024-05-09T05:01:45.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.600+0000 7fa9dcff9640 1 --2- 172.21.15.60:0/2663720785 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa9c00762d0 0x7fa9c0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.600+0000 7fa9dcff9640 1 -- 172.21.15.60:0/2663720785 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fa9d80976f0 con 0x7fa9e015f300 2024-05-09T05:01:45.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.600+0000 7fa9df7fe640 1 --2- 172.21.15.60:0/2663720785 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa9c00762d0 0x7fa9c0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.600+0000 7fa9e8df2740 1 Processor -- start 2024-05-09T05:01:45.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.601+0000 7fa9e8df2740 1 -- start start 2024-05-09T05:01:45.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.601+0000 7fa9df7fe640 1 --2- 172.21.15.60:0/2663720785 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa9c00762d0 0x7fa9c0078790 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0006fd0 tx=0x7fa9d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:45.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.601+0000 7fa9e8df2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787adfff0 0x562787ae2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.601+0000 7fa9e8df2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787ae03d0 0x562787ae2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.601+0000 7fa9e8df2740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562787ab2c50 0x562787ab6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.601+0000 7fa9e8df2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x562787a1c8d0 con 0x562787adfff0 2024-05-09T05:01:45.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.601+0000 7fa9e8df2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5627879d5c10 con 0x562787ae03d0 2024-05-09T05:01:45.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.601+0000 7fa9e8df2740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x562787a18890 con 0x562787ab2c50 2024-05-09T05:01:45.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.601+0000 7fa9df7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787adfff0 0x562787ae2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.601+0000 7fa9df7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787adfff0 0x562787ae2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37798/0 (socket says 172.21.15.60:37798) 2024-05-09T05:01:45.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.602+0000 7fa9df7fe640 1 -- 172.21.15.60:0/3116947803 learned_addr learned my addr 172.21.15.60:0/3116947803 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:45.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.602+0000 7fa9d7fff640 1 --2- 172.21.15.60:0/3116947803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787ae03d0 0x562787ae2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.602+0000 7fa9dffff640 1 --2- 172.21.15.60:0/3116947803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562787ab2c50 0x562787ab6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.602+0000 7fa9df7fe640 1 -- 172.21.15.60:0/3116947803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562787ab2c50 msgr2=0x562787ab6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:45.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.602+0000 7fa9df7fe640 1 --2- 172.21.15.60:0/3116947803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562787ab2c50 0x562787ab6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.602+0000 7fa9df7fe640 1 -- 172.21.15.60:0/3116947803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787ae03d0 msgr2=0x562787ae2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:45.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.602+0000 7fa9df7fe640 1 --2- 172.21.15.60:0/3116947803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787ae03d0 0x562787ae2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.602+0000 7fa9df7fe640 1 -- 172.21.15.60:0/3116947803 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627879dd5d0 con 0x562787adfff0 2024-05-09T05:01:45.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.602+0000 7fa9df7fe640 1 --2- 172.21.15.60:0/3116947803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787adfff0 0x562787ae2800 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000fa20 tx=0x7fa9d0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=effa1bec5a99e028 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:45.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.602+0000 7fa9d57fa640 1 -- 172.21.15.60:0/3116947803 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d001b050 con 0x562787adfff0 2024-05-09T05:01:45.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.602+0000 7fa9d57fa640 1 -- 172.21.15.60:0/3116947803 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9d0025070 con 0x562787adfff0 2024-05-09T05:01:45.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.602+0000 7fa9d57fa640 1 -- 172.21.15.60:0/3116947803 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0014de0 con 0x562787adfff0 2024-05-09T05:01:45.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.603+0000 7fa9e8df2740 1 -- 172.21.15.60:0/3116947803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787adfff0 msgr2=0x562787ae2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:45.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.603+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/3116947803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787adfff0 0x562787ae2800 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000fa20 tx=0x7fa9d0012000 comp rx=0 tx=0).stop 2024-05-09T05:01:45.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.603+0000 7fa9e8df2740 1 -- 172.21.15.60:0/3116947803 shutdown_connections 2024-05-09T05:01:45.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.603+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/3116947803 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562787ab2c50 0x562787ab6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.603+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/3116947803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787ae03d0 0x562787ae2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.603+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/3116947803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787adfff0 0x562787ae2800 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.603+0000 7fa9e8df2740 1 -- 172.21.15.60:0/3116947803 >> 172.21.15.60:0/3116947803 conn(0x562787ab4ec0 msgr2=0x562787afd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:45.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.603+0000 7fa9e8df2740 1 -- 172.21.15.60:0/3116947803 shutdown_connections 2024-05-09T05:01:45.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.603+0000 7fa9e8df2740 1 -- 172.21.15.60:0/3116947803 wait complete. 2024-05-09T05:01:45.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.603+0000 7fa9e8df2740 1 -- start start 2024-05-09T05:01:45.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.603+0000 7fa9e8df2740 1 Processor -- start 2024-05-09T05:01:45.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.604+0000 7fa9e8df2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787adfff0 0x562787aff900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.604+0000 7fa9e8df2740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562787ae03d0 0x562787affe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.604+0000 7fa9e8df2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787ad6de0 0x562787af9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.604+0000 7fa9e8df2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x562787ab8070 con 0x562787adfff0 2024-05-09T05:01:45.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.604+0000 7fa9e8df2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5627879d9760 con 0x562787ad6de0 2024-05-09T05:01:45.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.604+0000 7fa9e8df2740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x562787ab8350 con 0x562787ae03d0 2024-05-09T05:01:45.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.604+0000 7fa9e8df2740 10 client.0 _ll_register_callbacks cb 0x562787a12cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:01:45.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.604+0000 7fa9df7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787adfff0 0x562787aff900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.604+0000 7fa9df7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787adfff0 0x562787aff900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37812/0 (socket says 172.21.15.60:37812) 2024-05-09T05:01:45.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.604+0000 7fa9df7fe640 1 -- 172.21.15.60:0/1651486672 learned_addr learned my addr 172.21.15.60:0/1651486672 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:45.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.604+0000 7fa9d7fff640 1 --2- 172.21.15.60:0/1651486672 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562787ae03d0 0x562787affe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.604+0000 7fa9dffff640 1 --2- 172.21.15.60:0/1651486672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787ad6de0 0x562787af9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9df7fe640 1 -- 172.21.15.60:0/1651486672 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562787ae03d0 msgr2=0x562787affe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:45.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9df7fe640 1 --2- 172.21.15.60:0/1651486672 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562787ae03d0 0x562787affe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9df7fe640 1 -- 172.21.15.60:0/1651486672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787ad6de0 msgr2=0x562787af9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:45.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9df7fe640 1 --2- 172.21.15.60:0/1651486672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787ad6de0 0x562787af9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9df7fe640 1 -- 172.21.15.60:0/1651486672 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56278772e2d0 con 0x562787adfff0 2024-05-09T05:01:45.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9dffff640 1 --2- 172.21.15.60:0/1651486672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787ad6de0 0x562787af9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:45.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9df7fe640 1 --2- 172.21.15.60:0/1651486672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787adfff0 0x562787aff900 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000f5b0 tx=0x7fa9d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b697913457205d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:45.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9be7fc640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-05-09T05:01:45.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9be7fc640 1 -- 172.21.15.60:0/1651486672 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0020910 con 0x562787adfff0 2024-05-09T05:01:45.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9be7fc640 1 -- 172.21.15.60:0/1651486672 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fa9d0035c80 con 0x562787adfff0 2024-05-09T05:01:45.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9e8df2740 1 -- 172.21.15.60:0/1651486672 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9d0025070 con 0x562787adfff0 2024-05-09T05:01:45.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9be7fc640 1 -- 172.21.15.60:0/1651486672 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa9d001b020 con 0x562787adfff0 2024-05-09T05:01:45.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9e8df2740 10 client.5535 fetch_fsmap learned FSMap version 27 2024-05-09T05:01:45.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9e8df2740 1 -- 172.21.15.60:0/1651486672 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) -- 0x5627877f86b0 con 0x562787adfff0 2024-05-09T05:01:45.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9be7fc640 1 -- 172.21.15.60:0/1651486672 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9d0025070 con 0x562787adfff0 2024-05-09T05:01:45.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9be7fc640 1 -- 172.21.15.60:0/1651486672 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 27) ==== 3494+0+0 (secure 0 0 0) 0x7fa9d0032070 con 0x562787adfff0 2024-05-09T05:01:45.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9e8df2740 10 client.5535 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T05:01:45.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9e8df2740 10 client.5535 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T05:01:45.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9e8df2740 20 client.5535 populate_metadata read hostname 'smithi060' 2024-05-09T05:01:45.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/1651486672 >> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] conn(0x562787af8200 0x562787af85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9e8df2740 4 client.5535 mds_command: new command op to 5415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T05:01:45.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.605+0000 7fa9e8df2740 1 -- 172.21.15.60:0/1651486672 --> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627879a0760 con 0x562787af8200 2024-05-09T05:01:45.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.606+0000 7fa9d7fff640 1 --2- 172.21.15.60:0/1651486672 >> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] conn(0x562787af8200 0x562787af85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.606+0000 7fa9d7fff640 1 --2- 172.21.15.60:0/1651486672 >> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] conn(0x562787af8200 0x562787af85e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aca0329060314d88 server_cookie=e9bffb15cefe569e in_seq=0 out_seq=0 2024-05-09T05:01:45.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.606+0000 7fa9be7fc640 10 client.5535 ms_handle_connect on v2:172.21.15.60:6833/2425263408 2024-05-09T05:01:45.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.607+0000 7fa9be7fc640 1 -- 172.21.15.60:0/1651486672 <== mds.0 v2:172.21.15.60:6833/2425263408 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5627879a0760 con 0x562787af8200 2024-05-09T05:01:45.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.607+0000 7fa9be7fc640 10 client.5535 handle_command_reply: tid=0 2024-05-09T05:01:45.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.607+0000 7fa9e8df2740 1 client.5535 shutdown 2024-05-09T05:01:45.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.607+0000 7fa9e8df2740 1 -- 172.21.15.60:0/1651486672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787adfff0 msgr2=0x562787aff900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:45.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.607+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/1651486672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787adfff0 0x562787aff900 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000f5b0 tx=0x7fa9d0004730 comp rx=0 tx=0).stop 2024-05-09T05:01:45.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.607+0000 7fa9e8df2740 1 -- 172.21.15.60:0/1651486672 shutdown_connections 2024-05-09T05:01:45.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.607+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/1651486672 >> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] conn(0x562787af8200 0x562787af85e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.607+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/1651486672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787ad6de0 0x562787af9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.607+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/1651486672 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562787ae03d0 0x562787affe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.607+0000 7fa9be7fc640 0 client.5535 ms_handle_reset on v2:172.21.15.60:6833/2425263408 2024-05-09T05:01:45.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.607+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/1651486672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787adfff0 0x562787aff900 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.607+0000 7fa9e8df2740 1 -- 172.21.15.60:0/1651486672 >> 172.21.15.60:0/1651486672 conn(0x562787ab2c50 msgr2=0x562787ab1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:45.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.607+0000 7fa9e8df2740 1 -- 172.21.15.60:0/1651486672 shutdown_connections 2024-05-09T05:01:45.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.607+0000 7fa9e8df2740 1 -- 172.21.15.60:0/1651486672 wait complete. 2024-05-09T05:01:45.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.607+0000 7fa9e8df2740 20 client.5535 trim_cache size 0 max 16384 2024-05-09T05:01:45.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.626+0000 7fa9e8df2740 1 Processor -- start 2024-05-09T05:01:45.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.626+0000 7fa9e8df2740 1 -- start start 2024-05-09T05:01:45.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.626+0000 7fa9e8df2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787ad6de0 0x562787ab6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.626+0000 7fa9e8df2740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562787adfff0 0x562787ab69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.626+0000 7fa9e8df2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787ae03d0 0x562787ab3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.626+0000 7fa9e8df2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e0200f10 con 0x562787ad6de0 2024-05-09T05:01:45.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.626+0000 7fa9e8df2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e01f5fe0 con 0x562787ae03d0 2024-05-09T05:01:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.626+0000 7fa9e8df2740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa9e01f6630 con 0x562787adfff0 2024-05-09T05:01:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.627+0000 7fa9df7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787ad6de0 0x562787ab6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.627+0000 7fa9df7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787ad6de0 0x562787ab6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37818/0 (socket says 172.21.15.60:37818) 2024-05-09T05:01:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.627+0000 7fa9df7fe640 1 -- 172.21.15.60:0/2719048301 learned_addr learned my addr 172.21.15.60:0/2719048301 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.627+0000 7fa9df7fe640 1 -- 172.21.15.60:0/2719048301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562787adfff0 msgr2=0x562787ab69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.627+0000 7fa9df7fe640 1 --2- 172.21.15.60:0/2719048301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562787adfff0 0x562787ab69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.627+0000 7fa9df7fe640 1 -- 172.21.15.60:0/2719048301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787ae03d0 msgr2=0x562787ab3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.627+0000 7fa9df7fe640 1 --2- 172.21.15.60:0/2719048301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787ae03d0 0x562787ab3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.627+0000 7fa9df7fe640 1 -- 172.21.15.60:0/2719048301 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562787ad68c0 con 0x562787ad6de0 2024-05-09T05:01:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.627+0000 7fa9df7fe640 1 --2- 172.21.15.60:0/2719048301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787ad6de0 0x562787ab6460 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000f7d0 tx=0x7fa9d002d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7675e128104c4f51 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.627+0000 7fa9d4ff9640 1 -- 172.21.15.60:0/2719048301 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0020ad0 con 0x562787ad6de0 2024-05-09T05:01:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.627+0000 7fa9d4ff9640 1 -- 172.21.15.60:0/2719048301 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9d0025040 con 0x562787ad6de0 2024-05-09T05:01:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.627+0000 7fa9d4ff9640 1 -- 172.21.15.60:0/2719048301 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0005b90 con 0x562787ad6de0 2024-05-09T05:01:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.628+0000 7fa9e8df2740 1 -- 172.21.15.60:0/2719048301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787ad6de0 msgr2=0x562787ab6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:45.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.628+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/2719048301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787ad6de0 0x562787ab6460 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000f7d0 tx=0x7fa9d002d040 comp rx=0 tx=0).stop 2024-05-09T05:01:45.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.628+0000 7fa9e8df2740 1 -- 172.21.15.60:0/2719048301 shutdown_connections 2024-05-09T05:01:45.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.628+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/2719048301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787ae03d0 0x562787ab3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.628+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/2719048301 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562787adfff0 0x562787ab69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.628+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/2719048301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787ad6de0 0x562787ab6460 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.628+0000 7fa9e8df2740 1 -- 172.21.15.60:0/2719048301 >> 172.21.15.60:0/2719048301 conn(0x562787ab2c50 msgr2=0x562787ae6780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:45.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.628+0000 7fa9e8df2740 1 -- 172.21.15.60:0/2719048301 shutdown_connections 2024-05-09T05:01:45.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.628+0000 7fa9e8df2740 1 -- 172.21.15.60:0/2719048301 wait complete. 2024-05-09T05:01:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.628+0000 7fa9e8df2740 1 -- start start 2024-05-09T05:01:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.628+0000 7fa9e8df2740 1 Processor -- start 2024-05-09T05:01:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.629+0000 7fa9e8df2740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562787ad6de0 0x562787adf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.629+0000 7fa9e8df2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787adfff0 0x562787b792a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.629+0000 7fa9e8df2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787ae03d0 0x562787af9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.629+0000 7fa9e8df2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x562787af5dd0 con 0x562787ae03d0 2024-05-09T05:01:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.629+0000 7fa9e8df2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562787b015f0 con 0x562787adfff0 2024-05-09T05:01:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.629+0000 7fa9e8df2740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x562787afa0c0 con 0x562787ad6de0 2024-05-09T05:01:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.629+0000 7fa9e8df2740 10 client.0 _ll_register_callbacks cb 0x7fa9c8035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:01:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.629+0000 7fa9dffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787ae03d0 0x562787af9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.629+0000 7fa9dffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787ae03d0 0x562787af9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37832/0 (socket says 172.21.15.60:37832) 2024-05-09T05:01:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.629+0000 7fa9dffff640 1 -- 172.21.15.60:0/548211855 learned_addr learned my addr 172.21.15.60:0/548211855 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.629+0000 7fa9df7fe640 1 --2- 172.21.15.60:0/548211855 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562787ad6de0 0x562787adf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.629+0000 7fa9d7fff640 1 --2- 172.21.15.60:0/548211855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787adfff0 0x562787b792a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9dffff640 1 -- 172.21.15.60:0/548211855 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562787ad6de0 msgr2=0x562787adf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9dffff640 1 --2- 172.21.15.60:0/548211855 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562787ad6de0 0x562787adf040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9dffff640 1 -- 172.21.15.60:0/548211855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787adfff0 msgr2=0x562787b792a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9dffff640 1 --2- 172.21.15.60:0/548211855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787adfff0 0x562787b792a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9dffff640 1 -- 172.21.15.60:0/548211855 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562787af4190 con 0x562787ae03d0 2024-05-09T05:01:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9dffff640 1 --2- 172.21.15.60:0/548211855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787ae03d0 0x562787af9820 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa9d805cdb0 tx=0x7fa9d8060590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2957d423e4f26c1d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9be7fc640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-05-09T05:01:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9be7fc640 1 -- 172.21.15.60:0/548211855 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8079020 con 0x562787ae03d0 2024-05-09T05:01:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9be7fc640 1 -- 172.21.15.60:0/548211855 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7fa9d805fc20 con 0x562787ae03d0 2024-05-09T05:01:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9e8df2740 1 -- 172.21.15.60:0/548211855 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9d0025040 con 0x562787ae03d0 2024-05-09T05:01:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9be7fc640 1 -- 172.21.15.60:0/548211855 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa9d8067e60 con 0x562787ae03d0 2024-05-09T05:01:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9e8df2740 10 client.5541 fetch_fsmap learned FSMap version 27 2024-05-09T05:01:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9e8df2740 1 -- 172.21.15.60:0/548211855 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) -- 0x562787af9d60 con 0x562787ae03d0 2024-05-09T05:01:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9be7fc640 1 -- 172.21.15.60:0/548211855 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9d0025040 con 0x562787ae03d0 2024-05-09T05:01:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9be7fc640 1 -- 172.21.15.60:0/548211855 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 27) ==== 3494+0+0 (secure 0 0 0) 0x7fa9d8061af0 con 0x562787ae03d0 2024-05-09T05:01:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9e8df2740 10 client.5541 fetch_fsmap finished waiting for FSMap version 27 2024-05-09T05:01:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9e8df2740 10 client.5541 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T05:01:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9e8df2740 20 client.5541 populate_metadata read hostname 'smithi060' 2024-05-09T05:01:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/548211855 >> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] conn(0x562787afd800 0x562787afdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9e8df2740 4 client.5541 mds_command: new command op to 5415 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T05:01:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.630+0000 7fa9e8df2740 1 -- 172.21.15.60:0/548211855 --> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] -- command(tid 0: {"prefix": "status"}) -- 0x562787a2a4e0 con 0x562787afd800 2024-05-09T05:01:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.631+0000 7fa9df7fe640 1 --2- 172.21.15.60:0/548211855 >> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] conn(0x562787afd800 0x562787afdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.631+0000 7fa9be7fc640 10 client.5541 ms_handle_connect on v2:172.21.15.60:6833/2425263408 2024-05-09T05:01:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.631+0000 7fa9df7fe640 1 --2- 172.21.15.60:0/548211855 >> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] conn(0x562787afd800 0x562787afdbe0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74c53b722e70869f server_cookie=315e665693a88e71 in_seq=0 out_seq=0 2024-05-09T05:01:45.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.631+0000 7fa9be7fc640 1 -- 172.21.15.60:0/548211855 <== mds.0 v2:172.21.15.60:6833/2425263408 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x562787a2a4e0 con 0x562787afd800 2024-05-09T05:01:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.631+0000 7fa9be7fc640 10 client.5541 handle_command_reply: tid=0 2024-05-09T05:01:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.631+0000 7fa9e8df2740 1 client.5541 shutdown 2024-05-09T05:01:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.631+0000 7fa9e8df2740 1 -- 172.21.15.60:0/548211855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787ae03d0 msgr2=0x562787af9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.631+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/548211855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787ae03d0 0x562787af9820 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa9d805cdb0 tx=0x7fa9d8060590 comp rx=0 tx=0).stop 2024-05-09T05:01:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.631+0000 7fa9e8df2740 1 -- 172.21.15.60:0/548211855 shutdown_connections 2024-05-09T05:01:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.631+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/548211855 >> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] conn(0x562787afd800 0x562787afdbe0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.631+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/548211855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x562787ae03d0 0x562787af9820 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.631+0000 7fa9be7fc640 0 client.5541 ms_handle_reset on v2:172.21.15.60:6833/2425263408 2024-05-09T05:01:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.631+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/548211855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562787adfff0 0x562787b792a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.631+0000 7fa9e8df2740 1 --2- 172.21.15.60:0/548211855 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x562787ad6de0 0x562787adf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.631+0000 7fa9e8df2740 1 -- 172.21.15.60:0/548211855 >> 172.21.15.60:0/548211855 conn(0x562787ab2c50 msgr2=0x562787b21e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.631+0000 7fa9e8df2740 1 -- 172.21.15.60:0/548211855 shutdown_connections 2024-05-09T05:01:45.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.631+0000 7fa9e8df2740 1 -- 172.21.15.60:0/548211855 wait complete. 2024-05-09T05:01:45.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.632+0000 7fa9e8df2740 20 client.5541 trim_cache size 0 max 16384 2024-05-09T05:01:45.638 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:01:45.638 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2", 2024-05-09T05:01:45.638 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2024-05-09T05:01:45.638 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 5415, 2024-05-09T05:01:45.638 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2024-05-09T05:01:45.638 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2024-05-09T05:01:45.638 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2024-05-09T05:01:45.638 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 5.784341682, 2024-05-09T05:01:45.638 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 27, 2024-05-09T05:01:45.638 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 51, 2024-05-09T05:01:45.639 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 51, 2024-05-09T05:01:45.639 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 10.763292633000001 2024-05-09T05:01:45.639 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:01:45.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.632+0000 7fa9e60d3640 1 -- 172.21.15.60:0/2663720785 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa9c00762d0 msgr2=0x7fa9c0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.632+0000 7fa9e60d3640 1 --2- 172.21.15.60:0/2663720785 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa9c00762d0 0x7fa9c0078790 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0006fd0 tx=0x7fa9d0008040 comp rx=0 tx=0).stop 2024-05-09T05:01:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.632+0000 7fa9e60d3640 1 -- 172.21.15.60:0/2663720785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e015f300 msgr2=0x7fa9e015c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.632+0000 7fa9e60d3640 1 --2- 172.21.15.60:0/2663720785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e015f300 0x7fa9e015c280 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa9d800c7a0 tx=0x7fa9d800cc60 comp rx=0 tx=0).stop 2024-05-09T05:01:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.632+0000 7fa9e60d3640 1 -- 172.21.15.60:0/2663720785 shutdown_connections 2024-05-09T05:01:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.632+0000 7fa9e60d3640 1 --2- 172.21.15.60:0/2663720785 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa9c00762d0 0x7fa9c0078790 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.632+0000 7fa9e60d3640 1 --2- 172.21.15.60:0/2663720785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e015f300 0x7fa9e015c280 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.632+0000 7fa9e60d3640 1 --2- 172.21.15.60:0/2663720785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9e00d3930 0x7fa9e00aed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.632+0000 7fa9e60d3640 1 --2- 172.21.15.60:0/2663720785 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa9e006f9f0 0x7fa9e00ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.632+0000 7fa9e60d3640 1 -- 172.21.15.60:0/2663720785 >> 172.21.15.60:0/2663720785 conn(0x7fa9e009b850 msgr2=0x7fa9e009a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.632+0000 7fa9e60d3640 1 -- 172.21.15.60:0/2663720785 shutdown_connections 2024-05-09T05:01:45.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.632+0000 7fa9e60d3640 1 -- 172.21.15.60:0/2663720785 wait complete. 2024-05-09T05:01:45.644 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T05:01:45.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.747+0000 7f4180254640 1 Processor -- start 2024-05-09T05:01:45.747 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.747+0000 7f4180254640 1 -- start start 2024-05-09T05:01:45.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.748+0000 7f4180254640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f417809c8b0 0x7f417809e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.748+0000 7f4180254640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4178071f30 con 0x7f41780b4d40 2024-05-09T05:01:45.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.748+0000 7f4180254640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4178072210 con 0x7f417815f3e0 2024-05-09T05:01:45.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.748+0000 7f4180254640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f4178071ce0 con 0x7f417809c8b0 2024-05-09T05:01:45.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.748+0000 7f417e7f1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f417809c8b0 0x7f417809e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.748+0000 7f417e7f1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f417809c8b0 0x7f417809e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37842/0 (socket says 172.21.15.60:37842) 2024-05-09T05:01:45.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.748+0000 7f417d7ef640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f41780b4d40 0x7f417809e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57772/0 (socket says 172.21.15.60:57772) 2024-05-09T05:01:45.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.748+0000 7f417e7f1640 1 -- 172.21.15.60:0/634179524 learned_addr learned my addr 172.21.15.60:0/634179524 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:45.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.748+0000 7f417e7f1640 1 -- 172.21.15.60:0/634179524 >> v1:172.21.15.114:6789/0 conn(0x7f417815f3e0 legacy=0x7f41780b4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:45.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.748+0000 7f417e7f1640 1 -- 172.21.15.60:0/634179524 >> v1:172.21.15.60:6789/0 conn(0x7f41780b4d40 legacy=0x7f417809e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:45.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.748+0000 7f417e7f1640 1 -- 172.21.15.60:0/634179524 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41780724f0 con 0x7f417809c8b0 2024-05-09T05:01:45.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.748+0000 7f417cfee640 1 -- 172.21.15.60:0/634179524 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3286127632 0 0) 0x7f4178071f30 con 0x7f41780b4d40 2024-05-09T05:01:45.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.749+0000 7f417e7f1640 1 --2- 172.21.15.60:0/634179524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f417809c8b0 0x7f417809e810 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4178071e60 tx=0x7f417002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54eaee1bc3c285cc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:45.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.749+0000 7f417cfee640 1 -- 172.21.15.60:0/634179524 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4170036070 con 0x7f417809c8b0 2024-05-09T05:01:45.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.749+0000 7f417cfee640 1 -- 172.21.15.60:0/634179524 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4170031600 con 0x7f417809c8b0 2024-05-09T05:01:45.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.749+0000 7f417cfee640 1 -- 172.21.15.60:0/634179524 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41700319d0 con 0x7f417809c8b0 2024-05-09T05:01:45.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.750+0000 7f4180254640 1 -- 172.21.15.60:0/634179524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f417809c8b0 msgr2=0x7f417809e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:45.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.750+0000 7f4180254640 1 --2- 172.21.15.60:0/634179524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f417809c8b0 0x7f417809e810 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4178071e60 tx=0x7f417002a710 comp rx=0 tx=0).stop 2024-05-09T05:01:45.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.750+0000 7f4180254640 1 -- 172.21.15.60:0/634179524 shutdown_connections 2024-05-09T05:01:45.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.750+0000 7f4180254640 1 --2- 172.21.15.60:0/634179524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f417809c8b0 0x7f417809e810 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.750+0000 7f4180254640 1 -- 172.21.15.60:0/634179524 >> 172.21.15.60:0/634179524 conn(0x7f41780d1470 msgr2=0x7f41780d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:45.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.750+0000 7f4180254640 1 -- 172.21.15.60:0/634179524 shutdown_connections 2024-05-09T05:01:45.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.750+0000 7f4180254640 1 -- 172.21.15.60:0/634179524 wait complete. 2024-05-09T05:01:45.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.751+0000 7f4180254640 1 Processor -- start 2024-05-09T05:01:45.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.751+0000 7f4180254640 1 -- start start 2024-05-09T05:01:45.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.752+0000 7f4180254640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f417809c8b0 0x7f41781f50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.752+0000 7f4180254640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41780b4d40 0x7f41780ada30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.752+0000 7f4180254640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f417815f3e0 0x7f41781e79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.752+0000 7f4180254640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f417815f7c0 con 0x7f417809c8b0 2024-05-09T05:01:45.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.752+0000 7f4180254640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4178072ab0 con 0x7f41780b4d40 2024-05-09T05:01:45.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.752+0000 7f4180254640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4178072fd0 con 0x7f417815f3e0 2024-05-09T05:01:45.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.752+0000 7f417dff0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f417809c8b0 0x7f41781f50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.752+0000 7f417dff0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f417809c8b0 0x7f41781f50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37850/0 (socket says 172.21.15.60:37850) 2024-05-09T05:01:45.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.752+0000 7f417dff0640 1 -- 172.21.15.60:0/2207357379 learned_addr learned my addr 172.21.15.60:0/2207357379 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:45.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.752+0000 7f417e7f1640 1 --2- 172.21.15.60:0/2207357379 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f417815f3e0 0x7f41781e79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.752+0000 7f417d7ef640 1 --2- 172.21.15.60:0/2207357379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41780b4d40 0x7f41780ada30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.752+0000 7f417dff0640 1 -- 172.21.15.60:0/2207357379 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f417815f3e0 msgr2=0x7f41781e79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:45.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.752+0000 7f417dff0640 1 --2- 172.21.15.60:0/2207357379 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f417815f3e0 0x7f41781e79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.752+0000 7f417dff0640 1 -- 172.21.15.60:0/2207357379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41780b4d40 msgr2=0x7f41780ada30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:45.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.752+0000 7f417dff0640 1 --2- 172.21.15.60:0/2207357379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41780b4d40 0x7f41780ada30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.752+0000 7f417dff0640 1 -- 172.21.15.60:0/2207357379 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41780d37b0 con 0x7f417809c8b0 2024-05-09T05:01:45.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.752+0000 7f417dff0640 1 --2- 172.21.15.60:0/2207357379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f417809c8b0 0x7f41781f50a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f417400c3f0 tx=0x7f417400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:45.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.753+0000 7f416e7fc640 1 -- 172.21.15.60:0/2207357379 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4174011790 con 0x7f417809c8b0 2024-05-09T05:01:45.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.753+0000 7f416e7fc640 1 -- 172.21.15.60:0/2207357379 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4174017070 con 0x7f417809c8b0 2024-05-09T05:01:45.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.753+0000 7f4180254640 1 -- 172.21.15.60:0/2207357379 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41780d3a90 con 0x7f417809c8b0 2024-05-09T05:01:45.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.753+0000 7f416e7fc640 1 -- 172.21.15.60:0/2207357379 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4174011b90 con 0x7f417809c8b0 2024-05-09T05:01:45.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.753+0000 7f4180254640 1 -- 172.21.15.60:0/2207357379 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41780d4050 con 0x7f417809c8b0 2024-05-09T05:01:45.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.754+0000 7f4180254640 1 -- 172.21.15.60:0/2207357379 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4178051040 con 0x7f417809c8b0 2024-05-09T05:01:45.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.754+0000 7f416e7fc640 1 -- 172.21.15.60:0/2207357379 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f417401c460 con 0x7f417809c8b0 2024-05-09T05:01:45.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.755+0000 7f416e7fc640 1 --2- 172.21.15.60:0/2207357379 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f415c0762d0 0x7f415c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:45.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.755+0000 7f416e7fc640 1 -- 172.21.15.60:0/2207357379 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f41740975e0 con 0x7f417809c8b0 2024-05-09T05:01:45.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.755+0000 7f417d7ef640 1 --2- 172.21.15.60:0/2207357379 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f415c0762d0 0x7f415c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:45.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.755+0000 7f417d7ef640 1 --2- 172.21.15.60:0/2207357379 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f415c0762d0 0x7f415c078790 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f41780d2840 tx=0x7f4168007b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:45.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.759+0000 7f416e7fc640 1 -- 172.21.15.60:0/2207357379 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4174060db0 con 0x7f417809c8b0 2024-05-09T05:01:45.840 INFO:tasks.ceph.mds.c.smithi060.stderr:2024-05-09T05:01:45.839+0000 7f1c4abfe640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T05:01:45.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.991+0000 7f4180254640 1 -- 172.21.15.60:0/2207357379 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4178099c80 con 0x7f417809c8b0 2024-05-09T05:01:45.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.991+0000 7f416e7fc640 1 -- 172.21.15.60:0/2207357379 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+73 (secure 0 0 0) 0x7f41740606b0 con 0x7f417809c8b0 2024-05-09T05:01:45.992 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T05:01:45.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.993+0000 7f4180254640 1 -- 172.21.15.60:0/2207357379 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f415c0762d0 msgr2=0x7f415c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:45.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.993+0000 7f4180254640 1 --2- 172.21.15.60:0/2207357379 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f415c0762d0 0x7f415c078790 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f41780d2840 tx=0x7f4168007b50 comp rx=0 tx=0).stop 2024-05-09T05:01:45.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.993+0000 7f4180254640 1 -- 172.21.15.60:0/2207357379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f417809c8b0 msgr2=0x7f41781f50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:45.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.993+0000 7f4180254640 1 --2- 172.21.15.60:0/2207357379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f417809c8b0 0x7f41781f50a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f417400c3f0 tx=0x7f417400c8b0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.993+0000 7f4180254640 1 -- 172.21.15.60:0/2207357379 shutdown_connections 2024-05-09T05:01:45.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.993+0000 7f4180254640 1 --2- 172.21.15.60:0/2207357379 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f415c0762d0 0x7f415c078790 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.993+0000 7f4180254640 1 --2- 172.21.15.60:0/2207357379 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f417815f3e0 0x7f41781e79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.993+0000 7f4180254640 1 --2- 172.21.15.60:0/2207357379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41780b4d40 0x7f41780ada30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.993+0000 7f4180254640 1 --2- 172.21.15.60:0/2207357379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f417809c8b0 0x7f41781f50a0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:45.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.993+0000 7f4180254640 1 -- 172.21.15.60:0/2207357379 >> 172.21.15.60:0/2207357379 conn(0x7f41780d1470 msgr2=0x7f41780d1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:45.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.993+0000 7f4180254640 1 -- 172.21.15.60:0/2207357379 shutdown_connections 2024-05-09T05:01:45.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:45.993+0000 7f4180254640 1 -- 172.21.15.60:0/2207357379 wait complete. 2024-05-09T05:01:46.005 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-09T05:01:46.005 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T05:01:46.005 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2024-05-09T05:01:46.005 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T05:01:46.005 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T05:01:46.005 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-09T05:01:46.005 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-05-09T05:01:46.005 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T05:01:46.005 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T05:01:46.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:46.006 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2024-05-09T05:01:46.022 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T05:01:46.022 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T05:01:46.022 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T05:01:46.022 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:01:46.022 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2024-05-09T05:01:46.022 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:01:46.022 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:01:46.022 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2024-05-09T05:01:46.022 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2024-05-09T05:01:46.023 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:01:46.023 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2024-05-09T05:01:46.023 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2024-05-09T05:01:46.023 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T05:01:46.023 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:01:46.023 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2024-05-09T05:01:46.023 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T05:01:46.023 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 42658sec preferred_lft 42658sec 2024-05-09T05:01:46.023 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2024-05-09T05:01:46.023 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:01:46.023 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:01:46.023 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:01:46.023 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2024-05-09T05:01:46.023 INFO:teuthology.orchestra.run.smithi060.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T05:01:46.023 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 42:54:29:48:e0:84 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:01:46.023 INFO:teuthology.orchestra.run.smithi060.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T05:01:46.024 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:01:46.024 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::4889:3fff:fe4f:b9fb/64 scope link 2024-05-09T05:01:46.024 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:01:46.024 INFO:teuthology.orchestra.run.smithi060.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T05:01:46.024 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 42:54:29:48:e0:84 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T05:01:46.024 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::4054:29ff:fe48:e084/64 scope link 2024-05-09T05:01:46.024 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:01:46.024 INFO:teuthology.orchestra.run.smithi060.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T05:01:46.024 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 72:cf:f2:a5:7d:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T05:01:46.024 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::70cf:f2ff:fea5:7dba/64 scope link 2024-05-09T05:01:46.024 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:01:46.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:46.025 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T05:01:46.087 INFO:teuthology.orchestra.run.smithi060.stdout:1 2024-05-09T05:01:46.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:46.089 DEBUG:teuthology.orchestra.run.smithi060:> route 2024-05-09T05:01:46.147 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2024-05-09T05:01:46.147 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T05:01:46.147 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T05:01:46.147 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T05:01:46.147 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T05:01:46.147 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:01:46.147 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-05-09T05:01:46.147 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T05:01:46.148 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T05:01:46.148 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T05:01:46.148 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-05-09T05:01:46.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:01:46.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:01:46.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:46.278 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2024-05-09T05:01:46.294 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T05:01:46.294 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T05:01:46.294 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T05:01:46.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:46.295 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2024-05-09T05:01:46.360 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T05:01:46.360 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T05:01:46.416 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T05:01:46.416 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T05:01:46.417 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T05:01:46.472 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-05-09T05:01:46.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:46.472 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T05:01:46.557 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T05:01:46.557 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T05:01:46.572 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-05-09T05:01:46.572 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-05-09T05:01:46.572 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.114:3300,172.21.15.114:3301" 2024-05-09T05:01:46.572 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2.cephfs=/ 2024-05-09T05:01:46.572 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.60:3300/172.21.15.114:3300/172.21.15.114:3301" will pass to kernel 2024-05-09T05:01:46.572 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-09T05:01:46.573 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2024-05-09T05:01:46.573 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-09T05:01:46.573 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2024-05-09T05:01:46.573 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-09T05:01:46.575 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-09T05:01:46.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:46.575 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T05:01:46.601 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T05:01:46.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T05:01:46.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:01:46.706 INFO:teuthology.orchestra.run.smithi060.stdout:client4946 2024-05-09T05:01:46.706 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:01:46.707 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:01:46.723 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_extra1 --pg_num_min 64 2024-05-09T05:01:46.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.872+0000 7f431e30a640 1 Processor -- start 2024-05-09T05:01:46.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.872+0000 7f431e30a640 1 -- start start 2024-05-09T05:01:46.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.873+0000 7f431e30a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4318163300 0x7f43180b6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:46.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.873+0000 7f431e30a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43180d31b0 con 0x7f4318148120 2024-05-09T05:01:46.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.873+0000 7f431e30a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43180d3490 con 0x7f4318162e70 2024-05-09T05:01:46.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.873+0000 7f431e30a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f43180b5650 con 0x7f4318163300 2024-05-09T05:01:46.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.873+0000 7f431c8a7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4318163300 0x7f43180b6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:46.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.873+0000 7f4317fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4318148120 0x7f43180d2760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57778/0 (socket says 172.21.15.60:57778) 2024-05-09T05:01:46.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.873+0000 7f4317fff640 1 -- 172.21.15.60:0/365681321 learned_addr learned my addr 172.21.15.60:0/365681321 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:46.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.873+0000 7f431c8a7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4318163300 0x7f43180b6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37854/0 (socket says 172.21.15.60:37854) 2024-05-09T05:01:46.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.873+0000 7f431c8a7640 1 -- 172.21.15.60:0/365681321 >> v1:172.21.15.114:6789/0 conn(0x7f4318162e70 legacy=0x7f43180b5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:46.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.873+0000 7f431c8a7640 1 -- 172.21.15.60:0/365681321 >> v1:172.21.15.60:6789/0 conn(0x7f4318148120 legacy=0x7f43180d2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:46.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.873+0000 7f4316ffd640 1 -- 172.21.15.60:0/365681321 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2388500017 0 0) 0x7f43180d31b0 con 0x7f4318148120 2024-05-09T05:01:46.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.873+0000 7f431c8a7640 1 -- 172.21.15.60:0/365681321 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43180b5930 con 0x7f4318163300 2024-05-09T05:01:46.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.874+0000 7f431c8a7640 1 --2- 172.21.15.60:0/365681321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4318163300 0x7f43180b6560 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4310009d90 tx=0x7f431002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c540cfc52f2adda server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:46.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.874+0000 7f4316ffd640 1 -- 172.21.15.60:0/365681321 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4310007740 con 0x7f4318163300 2024-05-09T05:01:46.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.874+0000 7f4316ffd640 1 -- 172.21.15.60:0/365681321 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43100078e0 con 0x7f4318163300 2024-05-09T05:01:46.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.874+0000 7f4316ffd640 1 -- 172.21.15.60:0/365681321 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4310007cb0 con 0x7f4318163300 2024-05-09T05:01:46.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.875+0000 7f431e30a640 1 -- 172.21.15.60:0/365681321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4318163300 msgr2=0x7f43180b6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:46.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.875+0000 7f431e30a640 1 --2- 172.21.15.60:0/365681321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4318163300 0x7f43180b6560 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4310009d90 tx=0x7f431002a6f0 comp rx=0 tx=0).stop 2024-05-09T05:01:46.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.875+0000 7f431e30a640 1 -- 172.21.15.60:0/365681321 shutdown_connections 2024-05-09T05:01:46.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.875+0000 7f431e30a640 1 --2- 172.21.15.60:0/365681321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4318163300 0x7f43180b6560 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:46.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.875+0000 7f431e30a640 1 -- 172.21.15.60:0/365681321 >> 172.21.15.60:0/365681321 conn(0x7f431806f640 msgr2=0x7f4318073840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:46.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.875+0000 7f431e30a640 1 -- 172.21.15.60:0/365681321 shutdown_connections 2024-05-09T05:01:46.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.875+0000 7f431e30a640 1 -- 172.21.15.60:0/365681321 wait complete. 2024-05-09T05:01:46.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.876+0000 7f431e30a640 1 Processor -- start 2024-05-09T05:01:46.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.876+0000 7f431e30a640 1 -- start start 2024-05-09T05:01:46.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.877+0000 7f431e30a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4318148120 0x7f43180b4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:46.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.877+0000 7f431e30a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4318162e70 0x7f4318160750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:46.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.877+0000 7f431e30a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4318163300 0x7f4318072d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:46.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.877+0000 7f431e30a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f43181636e0 con 0x7f4318148120 2024-05-09T05:01:46.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.877+0000 7f431e30a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f43180d2e70 con 0x7f4318163300 2024-05-09T05:01:46.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.877+0000 7f431e30a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4318163b30 con 0x7f4318162e70 2024-05-09T05:01:46.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.877+0000 7f4317fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4318148120 0x7f43180b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:46.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.877+0000 7f4317fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4318148120 0x7f43180b4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37862/0 (socket says 172.21.15.60:37862) 2024-05-09T05:01:46.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.877+0000 7f4317fff640 1 -- 172.21.15.60:0/1758482341 learned_addr learned my addr 172.21.15.60:0/1758482341 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:46.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.877+0000 7f4317fff640 1 -- 172.21.15.60:0/1758482341 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4318162e70 msgr2=0x7f4318160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:46.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.877+0000 7f4317fff640 1 --2- 172.21.15.60:0/1758482341 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4318162e70 0x7f4318160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:46.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.877+0000 7f4317fff640 1 -- 172.21.15.60:0/1758482341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4318163300 msgr2=0x7f4318072d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:46.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.877+0000 7f4317fff640 1 --2- 172.21.15.60:0/1758482341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4318163300 0x7f4318072d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:46.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.877+0000 7f4317fff640 1 -- 172.21.15.60:0/1758482341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43180b46a0 con 0x7f4318148120 2024-05-09T05:01:46.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.877+0000 7f4317fff640 1 --2- 172.21.15.60:0/1758482341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4318148120 0x7f43180b4160 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f430800c450 tx=0x7f430800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:46.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.878+0000 7f4314ff9640 1 -- 172.21.15.60:0/1758482341 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4308018070 con 0x7f4318148120 2024-05-09T05:01:46.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.878+0000 7f4314ff9640 1 -- 172.21.15.60:0/1758482341 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43080074e0 con 0x7f4318148120 2024-05-09T05:01:46.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.878+0000 7f431e30a640 1 -- 172.21.15.60:0/1758482341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4318150a30 con 0x7f4318148120 2024-05-09T05:01:46.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.878+0000 7f4314ff9640 1 -- 172.21.15.60:0/1758482341 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4308004a20 con 0x7f4318148120 2024-05-09T05:01:46.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.878+0000 7f431e30a640 1 -- 172.21.15.60:0/1758482341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4318150ff0 con 0x7f4318148120 2024-05-09T05:01:46.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.879+0000 7f431e30a640 1 -- 172.21.15.60:0/1758482341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4318051040 con 0x7f4318148120 2024-05-09T05:01:46.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.879+0000 7f4314ff9640 1 -- 172.21.15.60:0/1758482341 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f430801b070 con 0x7f4318148120 2024-05-09T05:01:46.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.880+0000 7f4314ff9640 1 --2- 172.21.15.60:0/1758482341 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f43040762d0 0x7f4304078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:46.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.880+0000 7f4314ff9640 1 -- 172.21.15.60:0/1758482341 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 6101+0+0 (secure 0 0 0) 0x7f4308096ad0 con 0x7f4318148120 2024-05-09T05:01:46.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.880+0000 7f43177fe640 1 --2- 172.21.15.60:0/1758482341 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f43040762d0 0x7f4304078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:46.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.881+0000 7f43177fe640 1 --2- 172.21.15.60:0/1758482341 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f43040762d0 0x7f4304078790 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f431815f220 tx=0x7f430c007970 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:46.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:46.885+0000 7f4314ff9640 1 -- 172.21.15.60:0/1758482341 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4308060350 con 0x7f4318148120 2024-05-09T05:01:47.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:47.077+0000 7f431e30a640 1 -- 172.21.15.60:0/1758482341 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_extra1", "pg_num_min": 64} v 0) -- 0x7f4318162be0 con 0x7f4318148120 2024-05-09T05:01:47.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:47.899+0000 7f4314ff9640 1 -- 172.21.15.60:0/1758482341 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_extra1", "pg_num_min": 64}]=0 pool 'cephfs_data_extra1' created v52) ==== 144+0+0 (secure 0 0 0) 0x7f430805fc50 con 0x7f4318148120 2024-05-09T05:01:47.900 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data_extra1' created 2024-05-09T05:01:47.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:47.903+0000 7f431e30a640 1 -- 172.21.15.60:0/1758482341 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f43040762d0 msgr2=0x7f4304078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:47.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:47.903+0000 7f431e30a640 1 --2- 172.21.15.60:0/1758482341 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f43040762d0 0x7f4304078790 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f431815f220 tx=0x7f430c007970 comp rx=0 tx=0).stop 2024-05-09T05:01:47.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:47.903+0000 7f431e30a640 1 -- 172.21.15.60:0/1758482341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4318148120 msgr2=0x7f43180b4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:47.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:47.903+0000 7f431e30a640 1 --2- 172.21.15.60:0/1758482341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4318148120 0x7f43180b4160 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f430800c450 tx=0x7f430800c910 comp rx=0 tx=0).stop 2024-05-09T05:01:47.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:47.903+0000 7f431e30a640 1 -- 172.21.15.60:0/1758482341 shutdown_connections 2024-05-09T05:01:47.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:47.903+0000 7f431e30a640 1 --2- 172.21.15.60:0/1758482341 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f43040762d0 0x7f4304078790 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:47.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:47.903+0000 7f431e30a640 1 --2- 172.21.15.60:0/1758482341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4318163300 0x7f4318072d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:47.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:47.903+0000 7f431e30a640 1 --2- 172.21.15.60:0/1758482341 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4318162e70 0x7f4318160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:47.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:47.903+0000 7f431e30a640 1 --2- 172.21.15.60:0/1758482341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4318148120 0x7f43180b4160 secure :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f430800c450 tx=0x7f430800c910 comp rx=0 tx=0).stop 2024-05-09T05:01:47.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:47.903+0000 7f431e30a640 1 -- 172.21.15.60:0/1758482341 >> 172.21.15.60:0/1758482341 conn(0x7f431806f640 msgr2=0x7f43180b9630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:47.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:47.904+0000 7f431e30a640 1 -- 172.21.15.60:0/1758482341 shutdown_connections 2024-05-09T05:01:47.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:47.904+0000 7f431e30a640 1 -- 172.21.15.60:0/1758482341 wait complete. 2024-05-09T05:01:47.922 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_extra1 2024-05-09T05:01:48.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.033+0000 7f2903809640 1 Processor -- start 2024-05-09T05:01:48.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.033+0000 7f2903809640 1 -- start start 2024-05-09T05:01:48.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.034+0000 7f2903809640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f28fc1574c0 0x7f28fc161510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:48.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.034+0000 7f2903809640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28fc1578a0 0x7f28fc15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:48.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.034+0000 7f2903809640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28fc15e590 con 0x7f28fc15db30 2024-05-09T05:01:48.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.034+0000 7f2903809640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f28fc15dfd0 con 0x7f28fc1574c0 2024-05-09T05:01:48.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.034+0000 7f2903809640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f28fc15e2b0 con 0x7f28fc1578a0 2024-05-09T05:01:48.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.034+0000 7f2901da6640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f28fc15db30 0x7f28fc073a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57788/0 (socket says 172.21.15.60:57788) 2024-05-09T05:01:48.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.034+0000 7f2901da6640 1 -- 172.21.15.60:0/1634819324 learned_addr learned my addr 172.21.15.60:0/1634819324 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:48.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f29015a5640 1 --2- 172.21.15.60:0/1634819324 >> v2:172.21.15.114:3300/0 conn(0x7f28fc1574c0 0x7f28fc161510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:48.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f2900da4640 1 --2- 172.21.15.60:0/1634819324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28fc1578a0 0x7f28fc15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:48.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f28fbfff640 1 -- 172.21.15.60:0/1634819324 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3939465387 0 0) 0x7f28fc15e590 con 0x7f28fc15db30 2024-05-09T05:01:48.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f28fbfff640 1 -- 172.21.15.60:0/1634819324 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28e4003520 con 0x7f28fc15db30 2024-05-09T05:01:48.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f28fbfff640 1 -- 172.21.15.60:0/1634819324 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3337402318 0 0) 0x7f28e4003520 con 0x7f28fc15db30 2024-05-09T05:01:48.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f28fbfff640 1 -- 172.21.15.60:0/1634819324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28fc1578a0 msgr2=0x7f28fc15d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:48.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f28fbfff640 1 --2- 172.21.15.60:0/1634819324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28fc1578a0 0x7f28fc15d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:48.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f28fbfff640 1 -- 172.21.15.60:0/1634819324 >> v2:172.21.15.114:3300/0 conn(0x7f28fc1574c0 msgr2=0x7f28fc161510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:48.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f28fbfff640 1 --2- 172.21.15.60:0/1634819324 >> v2:172.21.15.114:3300/0 conn(0x7f28fc1574c0 0x7f28fc161510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:48.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f28fbfff640 1 -- 172.21.15.60:0/1634819324 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28fc160d10 con 0x7f28fc15db30 2024-05-09T05:01:48.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f28fbfff640 1 -- 172.21.15.60:0/1634819324 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f28ec001710 con 0x7f28fc15db30 2024-05-09T05:01:48.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f28fbfff640 1 -- 172.21.15.60:0/1634819324 >> v1:172.21.15.60:6789/0 conn(0x7f28fc15db30 legacy=0x7f28fc073a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:48.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f28fbfff640 1 --2- 172.21.15.60:0/1634819324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e4003a00 0x7f28e4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:48.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f28fbfff640 1 --2- 172.21.15.60:0/1634819324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28e4004800 0x7f28e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:48.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f28fbfff640 1 --2- 172.21.15.60:0/1634819324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28fc1578a0 0x7f28e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:48.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f28fbfff640 1 -- 172.21.15.60:0/1634819324 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc15dfd0 con 0x7f28e4003a00 2024-05-09T05:01:48.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f28fbfff640 1 -- 172.21.15.60:0/1634819324 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc15e2b0 con 0x7f28e4004800 2024-05-09T05:01:48.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f28fbfff640 1 -- 172.21.15.60:0/1634819324 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f28e400ba40 con 0x7f28fc1578a0 2024-05-09T05:01:48.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f29015a5640 1 --2- 172.21.15.60:0/1634819324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e4003a00 0x7f28e4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:48.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f2901da6640 1 --2- 172.21.15.60:0/1634819324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28fc1578a0 0x7f28e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:48.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.035+0000 7f2900da4640 1 --2- 172.21.15.60:0/1634819324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28e4004800 0x7f28e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:48.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.036+0000 7f29015a5640 1 -- 172.21.15.60:0/1634819324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28fc1578a0 msgr2=0x7f28e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:48.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.036+0000 7f29015a5640 1 --2- 172.21.15.60:0/1634819324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28fc1578a0 0x7f28e40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:48.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.036+0000 7f29015a5640 1 -- 172.21.15.60:0/1634819324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28e4004800 msgr2=0x7f28e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:48.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.036+0000 7f29015a5640 1 --2- 172.21.15.60:0/1634819324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28e4004800 0x7f28e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:48.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.036+0000 7f29015a5640 1 -- 172.21.15.60:0/1634819324 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e4003520 con 0x7f28e4003a00 2024-05-09T05:01:48.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.036+0000 7f29015a5640 1 --2- 172.21.15.60:0/1634819324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e4003a00 0x7f28e4003e20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f28f4009fc0 tx=0x7f28e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61cebfd211ed9546 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:48.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.036+0000 7f28fbfff640 1 -- 172.21.15.60:0/1634819324 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f4009c10 con 0x7f28e4003a00 2024-05-09T05:01:48.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.036+0000 7f28fbfff640 1 -- 172.21.15.60:0/1634819324 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28f4036070 con 0x7f28e4003a00 2024-05-09T05:01:48.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.036+0000 7f28fbfff640 1 -- 172.21.15.60:0/1634819324 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f402d070 con 0x7f28e4003a00 2024-05-09T05:01:48.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.037+0000 7f2903809640 1 -- 172.21.15.60:0/1634819324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e4003a00 msgr2=0x7f28e4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:48.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.037+0000 7f2903809640 1 --2- 172.21.15.60:0/1634819324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e4003a00 0x7f28e4003e20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f28f4009fc0 tx=0x7f28e4005830 comp rx=0 tx=0).stop 2024-05-09T05:01:48.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.037+0000 7f2901da6640 1 -- 172.21.15.60:0/1634819324 reap_dead start 2024-05-09T05:01:48.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.037+0000 7f2903809640 1 -- 172.21.15.60:0/1634819324 shutdown_connections 2024-05-09T05:01:48.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.037+0000 7f2903809640 1 -- 172.21.15.60:0/1634819324 >> 172.21.15.60:0/1634819324 conn(0x7f28fc0b4220 msgr2=0x7f28fc0b4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:48.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.037+0000 7f2903809640 1 -- 172.21.15.60:0/1634819324 shutdown_connections 2024-05-09T05:01:48.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.037+0000 7f2903809640 1 -- 172.21.15.60:0/1634819324 wait complete. 2024-05-09T05:01:48.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.038+0000 7f2903809640 1 Processor -- start 2024-05-09T05:01:48.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.038+0000 7f2903809640 1 -- start start 2024-05-09T05:01:48.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.039+0000 7f2903809640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e4003a00 0x7f28fc15b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:48.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.039+0000 7f2903809640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28fc0ab230 0x7f28fc1f6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:48.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.039+0000 7f2903809640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28fc1f6e70 0x7f28fc1f3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:48.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.039+0000 7f2903809640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc0b29f0 con 0x7f28e4003a00 2024-05-09T05:01:48.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.039+0000 7f2903809640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc161a50 con 0x7f28fc1f6e70 2024-05-09T05:01:48.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.039+0000 7f2903809640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f28fc0b2b70 con 0x7f28fc0ab230 2024-05-09T05:01:48.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.039+0000 7f29015a5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e4003a00 0x7f28fc15b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:48.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.039+0000 7f29015a5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e4003a00 0x7f28fc15b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37886/0 (socket says 172.21.15.60:37886) 2024-05-09T05:01:48.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.039+0000 7f29015a5640 1 -- 172.21.15.60:0/3286671696 learned_addr learned my addr 172.21.15.60:0/3286671696 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:48.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.039+0000 7f2901da6640 1 --2- 172.21.15.60:0/3286671696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28fc1f6e70 0x7f28fc1f3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:48.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.039+0000 7f2900da4640 1 --2- 172.21.15.60:0/3286671696 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28fc0ab230 0x7f28fc1f6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:48.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.040+0000 7f29015a5640 1 -- 172.21.15.60:0/3286671696 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28fc0ab230 msgr2=0x7f28fc1f6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:48.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.040+0000 7f29015a5640 1 --2- 172.21.15.60:0/3286671696 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28fc0ab230 0x7f28fc1f6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:48.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.040+0000 7f29015a5640 1 -- 172.21.15.60:0/3286671696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28fc1f6e70 msgr2=0x7f28fc1f3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:48.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.040+0000 7f29015a5640 1 --2- 172.21.15.60:0/3286671696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28fc1f6e70 0x7f28fc1f3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:48.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.040+0000 7f29015a5640 1 -- 172.21.15.60:0/3286671696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28fc15be00 con 0x7f28e4003a00 2024-05-09T05:01:48.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.040+0000 7f29015a5640 1 --2- 172.21.15.60:0/3286671696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e4003a00 0x7f28fc15b500 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f28f400e230 tx=0x7f28f4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:48.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.040+0000 7f28f9ffb640 1 -- 172.21.15.60:0/3286671696 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f4009c10 con 0x7f28e4003a00 2024-05-09T05:01:48.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.040+0000 7f28f9ffb640 1 -- 172.21.15.60:0/3286671696 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28f4003bb0 con 0x7f28e4003a00 2024-05-09T05:01:48.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.040+0000 7f2903809640 1 -- 172.21.15.60:0/3286671696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28fc15c0e0 con 0x7f28e4003a00 2024-05-09T05:01:48.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.040+0000 7f28f9ffb640 1 -- 172.21.15.60:0/3286671696 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f4004020 con 0x7f28e4003a00 2024-05-09T05:01:48.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.040+0000 7f2903809640 1 -- 172.21.15.60:0/3286671696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28fc15c6a0 con 0x7f28e4003a00 2024-05-09T05:01:48.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.041+0000 7f2903809640 1 -- 172.21.15.60:0/3286671696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28fc051040 con 0x7f28e4003a00 2024-05-09T05:01:48.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.042+0000 7f28f9ffb640 1 -- 172.21.15.60:0/3286671696 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f28f4005d10 con 0x7f28e4003a00 2024-05-09T05:01:48.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.042+0000 7f28f9ffb640 1 --2- 172.21.15.60:0/3286671696 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f28e80762d0 0x7f28e8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:48.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.043+0000 7f28f9ffb640 1 -- 172.21.15.60:0/3286671696 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6474+0+0 (secure 0 0 0) 0x7f28f40b8d90 con 0x7f28e4003a00 2024-05-09T05:01:48.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.043+0000 7f2900da4640 1 --2- 172.21.15.60:0/3286671696 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f28e80762d0 0x7f28e8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:48.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.043+0000 7f2900da4640 1 --2- 172.21.15.60:0/3286671696 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f28e80762d0 0x7f28e8078790 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f28fc15d070 tx=0x7f28f0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:48.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.047+0000 7f28f9ffb640 1 -- 172.21.15.60:0/3286671696 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f28f4082350 con 0x7f28e4003a00 2024-05-09T05:01:48.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.276+0000 7f2903809640 1 -- 172.21.15.60:0/3286671696 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_extra1"} v 0) -- 0x7f28fc06f130 con 0x7f28e4003a00 2024-05-09T05:01:48.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.901+0000 7f28f9ffb640 1 -- 172.21.15.60:0/3286671696 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_extra1"}]=0 added data pool 10 to fsmap v28) ==== 142+0+0 (secure 0 0 0) 0x7f28f408a340 con 0x7f28e4003a00 2024-05-09T05:01:48.902 INFO:teuthology.orchestra.run.smithi060.stderr:added data pool 10 to fsmap 2024-05-09T05:01:48.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.905+0000 7f2903809640 1 -- 172.21.15.60:0/3286671696 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f28e80762d0 msgr2=0x7f28e8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:48.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.905+0000 7f2903809640 1 --2- 172.21.15.60:0/3286671696 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f28e80762d0 0x7f28e8078790 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f28fc15d070 tx=0x7f28f0008040 comp rx=0 tx=0).stop 2024-05-09T05:01:48.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.906+0000 7f2903809640 1 -- 172.21.15.60:0/3286671696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e4003a00 msgr2=0x7f28fc15b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:48.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.906+0000 7f2903809640 1 --2- 172.21.15.60:0/3286671696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e4003a00 0x7f28fc15b500 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f28f400e230 tx=0x7f28f4009750 comp rx=0 tx=0).stop 2024-05-09T05:01:48.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.906+0000 7f2903809640 1 -- 172.21.15.60:0/3286671696 shutdown_connections 2024-05-09T05:01:48.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.906+0000 7f2903809640 1 --2- 172.21.15.60:0/3286671696 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f28e80762d0 0x7f28e8078790 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:48.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.906+0000 7f2903809640 1 --2- 172.21.15.60:0/3286671696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28fc1f6e70 0x7f28fc1f3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:48.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.906+0000 7f2903809640 1 --2- 172.21.15.60:0/3286671696 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f28fc0ab230 0x7f28fc1f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:48.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.906+0000 7f2903809640 1 --2- 172.21.15.60:0/3286671696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e4003a00 0x7f28fc15b500 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:48.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.906+0000 7f2903809640 1 -- 172.21.15.60:0/3286671696 >> 172.21.15.60:0/3286671696 conn(0x7f28fc0b4220 msgr2=0x7f28fc15f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:48.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.909+0000 7f2903809640 1 -- 172.21.15.60:0/3286671696 shutdown_connections 2024-05-09T05:01:48.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:48.911+0000 7f2903809640 1 -- 172.21.15.60:0/3286671696 wait complete. 2024-05-09T05:01:48.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:49.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.042+0000 7f219addd640 1 Processor -- start 2024-05-09T05:01:49.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.042+0000 7f219addd640 1 -- start start 2024-05-09T05:01:49.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.042+0000 7f219addd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21941606f0 0x7f21940d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.042+0000 7f219addd640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f219406ff60 con 0x7f219406f9f0 2024-05-09T05:01:49.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.042+0000 7f219addd640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2194070240 con 0x7f219415f300 2024-05-09T05:01:49.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.042+0000 7f219addd640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f2194070520 con 0x7f21941606f0 2024-05-09T05:01:49.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.043+0000 7f219937a640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f219406f9f0 0x7f219406ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57122/0 (socket says 172.21.15.60:57122) 2024-05-09T05:01:49.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.043+0000 7f219937a640 1 -- 172.21.15.60:0/2052735435 learned_addr learned my addr 172.21.15.60:0/2052735435 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:49.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.043+0000 7f2193fff640 1 --2- 172.21.15.60:0/2052735435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21941606f0 0x7f21940d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.043+0000 7f21937fe640 1 -- 172.21.15.60:0/2052735435 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2463931916 0 0) 0x7f219406ff60 con 0x7f219406f9f0 2024-05-09T05:01:49.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.043+0000 7f21937fe640 1 -- 172.21.15.60:0/2052735435 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2180003520 con 0x7f219406f9f0 2024-05-09T05:01:49.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.043+0000 7f21937fe640 1 -- 172.21.15.60:0/2052735435 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1718335387 0 0) 0x7f2194070240 con 0x7f219415f300 2024-05-09T05:01:49.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.043+0000 7f21937fe640 1 -- 172.21.15.60:0/2052735435 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f219406ff60 con 0x7f219415f300 2024-05-09T05:01:49.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.043+0000 7f21937fe640 1 -- 172.21.15.60:0/2052735435 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2685837824 0 0) 0x7f2180003520 con 0x7f219406f9f0 2024-05-09T05:01:49.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.043+0000 7f2193fff640 1 -- 172.21.15.60:0/2052735435 >> v1:172.21.15.114:6789/0 conn(0x7f219415f300 legacy=0x7f21941639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.043+0000 7f2193fff640 1 -- 172.21.15.60:0/2052735435 >> v1:172.21.15.60:6789/0 conn(0x7f219406f9f0 legacy=0x7f219406ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.043+0000 7f2193fff640 1 -- 172.21.15.60:0/2052735435 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f219406e740 con 0x7f21941606f0 2024-05-09T05:01:49.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.044+0000 7f2193fff640 1 --2- 172.21.15.60:0/2052735435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21941606f0 0x7f21940d33f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f217c0097d0 tx=0x7f217c02a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e78295d3be6e493e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:49.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.044+0000 7f21937fe640 1 -- 172.21.15.60:0/2052735435 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217c037070 con 0x7f21941606f0 2024-05-09T05:01:49.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.044+0000 7f21937fe640 1 -- 172.21.15.60:0/2052735435 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f217c02ac30 con 0x7f21941606f0 2024-05-09T05:01:49.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.044+0000 7f21937fe640 1 -- 172.21.15.60:0/2052735435 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2184001a70 con 0x7f21941606f0 2024-05-09T05:01:49.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.045+0000 7f219addd640 1 -- 172.21.15.60:0/2052735435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21941606f0 msgr2=0x7f21940d33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.045+0000 7f219addd640 1 --2- 172.21.15.60:0/2052735435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21941606f0 0x7f21940d33f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f217c0097d0 tx=0x7f217c02a1a0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.045+0000 7f219addd640 1 -- 172.21.15.60:0/2052735435 shutdown_connections 2024-05-09T05:01:49.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.045+0000 7f219addd640 1 --2- 172.21.15.60:0/2052735435 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f21941606f0 0x7f21940d33f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.045+0000 7f219addd640 1 -- 172.21.15.60:0/2052735435 >> 172.21.15.60:0/2052735435 conn(0x7f219409b850 msgr2=0x7f219409bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:49.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.045+0000 7f219addd640 1 -- 172.21.15.60:0/2052735435 shutdown_connections 2024-05-09T05:01:49.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.045+0000 7f219addd640 1 -- 172.21.15.60:0/2052735435 wait complete. 2024-05-09T05:01:49.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.046+0000 7f219addd640 1 Processor -- start 2024-05-09T05:01:49.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.046+0000 7f219addd640 1 -- start start 2024-05-09T05:01:49.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.047+0000 7f219addd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f219406f9f0 0x7f21940b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.047+0000 7f219addd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f219415f300 0x7f21940afb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.047+0000 7f219addd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21941606f0 0x7f21941f4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.047+0000 7f219addd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f219406f7d0 con 0x7f219415f300 2024-05-09T05:01:49.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.047+0000 7f219addd640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f219406f650 con 0x7f21941606f0 2024-05-09T05:01:49.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.047+0000 7f219addd640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f21940d3930 con 0x7f219406f9f0 2024-05-09T05:01:49.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.047+0000 7f2193fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f219415f300 0x7f21940afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.047+0000 7f2193fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f219415f300 0x7f21940afb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35554/0 (socket says 172.21.15.60:35554) 2024-05-09T05:01:49.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.047+0000 7f2193fff640 1 -- 172.21.15.60:0/404335670 learned_addr learned my addr 172.21.15.60:0/404335670 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:49.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.047+0000 7f219937a640 1 --2- 172.21.15.60:0/404335670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21941606f0 0x7f21941f4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.047+0000 7f2198b79640 1 --2- 172.21.15.60:0/404335670 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f219406f9f0 0x7f21940b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.047+0000 7f2193fff640 1 -- 172.21.15.60:0/404335670 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f219406f9f0 msgr2=0x7f21940b45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.047+0000 7f2193fff640 1 --2- 172.21.15.60:0/404335670 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f219406f9f0 0x7f21940b45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.047+0000 7f2193fff640 1 -- 172.21.15.60:0/404335670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21941606f0 msgr2=0x7f21941f4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.047+0000 7f2193fff640 1 --2- 172.21.15.60:0/404335670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21941606f0 0x7f21941f4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.047+0000 7f2193fff640 1 -- 172.21.15.60:0/404335670 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21940cfc10 con 0x7f219415f300 2024-05-09T05:01:49.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.048+0000 7f2193fff640 1 --2- 172.21.15.60:0/404335670 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f219415f300 0x7f21940afb70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f217c0097d0 tx=0x7f217c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:49.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.048+0000 7f21917fa640 1 -- 172.21.15.60:0/404335670 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217c037050 con 0x7f219415f300 2024-05-09T05:01:49.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.048+0000 7f21917fa640 1 -- 172.21.15.60:0/404335670 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f217c038040 con 0x7f219415f300 2024-05-09T05:01:49.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.048+0000 7f219addd640 1 -- 172.21.15.60:0/404335670 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21940b20d0 con 0x7f219415f300 2024-05-09T05:01:49.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.048+0000 7f21917fa640 1 -- 172.21.15.60:0/404335670 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217c006e10 con 0x7f219415f300 2024-05-09T05:01:49.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.048+0000 7f219addd640 1 -- 172.21.15.60:0/404335670 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21940b2690 con 0x7f219415f300 2024-05-09T05:01:49.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.049+0000 7f219addd640 1 -- 172.21.15.60:0/404335670 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2194051040 con 0x7f219415f300 2024-05-09T05:01:49.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.050+0000 7f21917fa640 1 -- 172.21.15.60:0/404335670 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f217c045020 con 0x7f219415f300 2024-05-09T05:01:49.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.050+0000 7f21917fa640 1 --2- 172.21.15.60:0/404335670 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f21740762d0 0x7f2174078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.050+0000 7f21917fa640 1 -- 172.21.15.60:0/404335670 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6506+0+0 (secure 0 0 0) 0x7f217c0b7800 con 0x7f219415f300 2024-05-09T05:01:49.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.050+0000 7f2198b79640 1 --2- 172.21.15.60:0/404335670 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f21740762d0 0x7f2174078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.051+0000 7f2198b79640 1 --2- 172.21.15.60:0/404335670 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f21740762d0 0x7f2174078790 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f218c001810 tx=0x7f218c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:49.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.054+0000 7f21917fa640 1 -- 172.21.15.60:0/404335670 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f217c04c020 con 0x7f219415f300 2024-05-09T05:01:49.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.286+0000 7f219addd640 1 -- 172.21.15.60:0/404335670 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f219409f8e0 con 0x7f219415f300 2024-05-09T05:01:49.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.286+0000 7f21917fa640 1 -- 172.21.15.60:0/404335670 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5190 (secure 0 0 0) 0x7f217c080800 con 0x7f219415f300 2024-05-09T05:01:49.287 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:49.287 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3604145570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3604145570},{"type":"v1","addr":"172.21.15.60:6834","nonce":3604145570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3949194455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3949194455},{"type":"v1","addr":"172.21.15.114:6836","nonce":3949194455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3288630382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3288630382},{"type":"v1","addr":"172.21.15.114:6837","nonce":3288630382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:39.237498+0000","modified":"2024-05-09T05:01:48.278623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5415":{"gid":5415,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2425263408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2425263408},{"type":"v1","addr":"172.21.15.60:6835","nonce":2425263408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9,10],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5415,"qdb_cluster":[5415]},"id":3}]} 2024-05-09T05:01:49.287 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 28 2024-05-09T05:01:49.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.288+0000 7f219addd640 1 -- 172.21.15.60:0/404335670 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f21740762d0 msgr2=0x7f2174078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.288+0000 7f219addd640 1 --2- 172.21.15.60:0/404335670 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f21740762d0 0x7f2174078790 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f218c001810 tx=0x7f218c009040 comp rx=0 tx=0).stop 2024-05-09T05:01:49.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.288+0000 7f219addd640 1 -- 172.21.15.60:0/404335670 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f219415f300 msgr2=0x7f21940afb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.288+0000 7f219addd640 1 --2- 172.21.15.60:0/404335670 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f219415f300 0x7f21940afb70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f217c0097d0 tx=0x7f217c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.289+0000 7f219addd640 1 -- 172.21.15.60:0/404335670 shutdown_connections 2024-05-09T05:01:49.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.289+0000 7f219addd640 1 --2- 172.21.15.60:0/404335670 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f21740762d0 0x7f2174078790 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.289+0000 7f219addd640 1 --2- 172.21.15.60:0/404335670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21941606f0 0x7f21941f4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.289+0000 7f219addd640 1 --2- 172.21.15.60:0/404335670 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f219415f300 0x7f21940afb70 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.289+0000 7f219addd640 1 --2- 172.21.15.60:0/404335670 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f219406f9f0 0x7f21940b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.289+0000 7f219addd640 1 -- 172.21.15.60:0/404335670 >> 172.21.15.60:0/404335670 conn(0x7f219409b850 msgr2=0x7f219409a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:49.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.289+0000 7f219addd640 1 -- 172.21.15.60:0/404335670 shutdown_connections 2024-05-09T05:01:49.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.289+0000 7f219addd640 1 -- 172.21.15.60:0/404335670 wait complete. 2024-05-09T05:01:49.300 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:01:49.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.404+0000 7fc243f40640 1 Processor -- start 2024-05-09T05:01:49.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.404+0000 7fc243f40640 1 -- start start 2024-05-09T05:01:49.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.405+0000 7fc243f40640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc23c073de0 0x7fc23c0702d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.405+0000 7fc243f40640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc23c0d08e0 con 0x7fc23c06fe20 2024-05-09T05:01:49.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.405+0000 7fc243f40640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc23c0d0bc0 con 0x7fc23c15f300 2024-05-09T05:01:49.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.405+0000 7fc243f40640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fc23c073240 con 0x7fc23c073de0 2024-05-09T05:01:49.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.405+0000 7fc2414db640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc23c06fe20 0x7fc23c0736d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57124/0 (socket says 172.21.15.60:57124) 2024-05-09T05:01:49.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.405+0000 7fc2414db640 1 -- 172.21.15.60:0/372295492 learned_addr learned my addr 172.21.15.60:0/372295492 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:49.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.405+0000 7fc2424dd640 1 --2- 172.21.15.60:0/372295492 >> v2:172.21.15.114:3300/0 conn(0x7fc23c073de0 0x7fc23c0702d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.405+0000 7fc240cda640 1 -- 172.21.15.60:0/372295492 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 276675463 0 0) 0x7fc23c0d08e0 con 0x7fc23c06fe20 2024-05-09T05:01:49.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.405+0000 7fc240cda640 1 -- 172.21.15.60:0/372295492 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc228003520 con 0x7fc23c06fe20 2024-05-09T05:01:49.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.405+0000 7fc2424dd640 1 -- 172.21.15.60:0/372295492 >> v1:172.21.15.114:6789/0 conn(0x7fc23c15f300 legacy=0x7fc23c072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.405+0000 7fc2424dd640 1 -- 172.21.15.60:0/372295492 >> v1:172.21.15.60:6789/0 conn(0x7fc23c06fe20 legacy=0x7fc23c0736d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.405+0000 7fc240cda640 1 -- 172.21.15.60:0/372295492 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2690376975 0 0) 0x7fc23c0d0bc0 con 0x7fc23c15f300 2024-05-09T05:01:49.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.405+0000 7fc2424dd640 1 -- 172.21.15.60:0/372295492 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc23c072ed0 con 0x7fc23c073de0 2024-05-09T05:01:49.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.406+0000 7fc2424dd640 1 --2- 172.21.15.60:0/372295492 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc23c073de0 0x7fc23c0702d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc234009d90 tx=0x7fc23402a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52606801a5183007 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:49.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.406+0000 7fc240cda640 1 -- 172.21.15.60:0/372295492 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc234007740 con 0x7fc23c073de0 2024-05-09T05:01:49.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.406+0000 7fc240cda640 1 -- 172.21.15.60:0/372295492 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2340078e0 con 0x7fc23c073de0 2024-05-09T05:01:49.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.406+0000 7fc240cda640 1 -- 172.21.15.60:0/372295492 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc234007cb0 con 0x7fc23c073de0 2024-05-09T05:01:49.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.407+0000 7fc243f40640 1 -- 172.21.15.60:0/372295492 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc23c073de0 msgr2=0x7fc23c0702d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.407+0000 7fc243f40640 1 --2- 172.21.15.60:0/372295492 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc23c073de0 0x7fc23c0702d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc234009d90 tx=0x7fc23402a6f0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.407+0000 7fc243f40640 1 -- 172.21.15.60:0/372295492 shutdown_connections 2024-05-09T05:01:49.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.407+0000 7fc243f40640 1 --2- 172.21.15.60:0/372295492 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc23c073de0 0x7fc23c0702d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.407+0000 7fc243f40640 1 -- 172.21.15.60:0/372295492 >> 172.21.15.60:0/372295492 conn(0x7fc23c0b4ef0 msgr2=0x7fc23c06e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:49.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.407+0000 7fc243f40640 1 -- 172.21.15.60:0/372295492 shutdown_connections 2024-05-09T05:01:49.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.407+0000 7fc243f40640 1 -- 172.21.15.60:0/372295492 wait complete. 2024-05-09T05:01:49.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.408+0000 7fc243f40640 1 Processor -- start 2024-05-09T05:01:49.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.408+0000 7fc243f40640 1 -- start start 2024-05-09T05:01:49.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.409+0000 7fc243f40640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc23c06fe20 0x7fc23c099fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.409+0000 7fc243f40640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc23c073de0 0x7fc23c1f4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.409+0000 7fc243f40640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c15f300 0x7fc23c0d3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.409+0000 7fc243f40640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc23c0d0670 con 0x7fc23c15f300 2024-05-09T05:01:49.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.409+0000 7fc243f40640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc23c0d04f0 con 0x7fc23c073de0 2024-05-09T05:01:49.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.409+0000 7fc243f40640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc23c15f6e0 con 0x7fc23c06fe20 2024-05-09T05:01:49.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.409+0000 7fc2424dd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c15f300 0x7fc23c0d3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.409+0000 7fc2424dd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c15f300 0x7fc23c0d3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35556/0 (socket says 172.21.15.60:35556) 2024-05-09T05:01:49.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.409+0000 7fc2424dd640 1 -- 172.21.15.60:0/3545036982 learned_addr learned my addr 172.21.15.60:0/3545036982 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:49.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.409+0000 7fc2414db640 1 --2- 172.21.15.60:0/3545036982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc23c073de0 0x7fc23c1f4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.409+0000 7fc241cdc640 1 --2- 172.21.15.60:0/3545036982 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc23c06fe20 0x7fc23c099fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.409+0000 7fc2424dd640 1 -- 172.21.15.60:0/3545036982 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc23c06fe20 msgr2=0x7fc23c099fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.409+0000 7fc2424dd640 1 --2- 172.21.15.60:0/3545036982 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc23c06fe20 0x7fc23c099fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.409+0000 7fc2424dd640 1 -- 172.21.15.60:0/3545036982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc23c073de0 msgr2=0x7fc23c1f4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.409+0000 7fc2424dd640 1 --2- 172.21.15.60:0/3545036982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc23c073de0 0x7fc23c1f4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.409+0000 7fc2424dd640 1 -- 172.21.15.60:0/3545036982 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc23c1f5680 con 0x7fc23c15f300 2024-05-09T05:01:49.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.409+0000 7fc2414db640 1 --2- 172.21.15.60:0/3545036982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc23c073de0 0x7fc23c1f4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:49.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.409+0000 7fc241cdc640 1 --2- 172.21.15.60:0/3545036982 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc23c06fe20 0x7fc23c099fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:49.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.410+0000 7fc2424dd640 1 --2- 172.21.15.60:0/3545036982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c15f300 0x7fc23c0d3940 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc234009d90 tx=0x7fc234004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:49.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.410+0000 7fc2327fc640 1 -- 172.21.15.60:0/3545036982 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc234040070 con 0x7fc23c15f300 2024-05-09T05:01:49.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.410+0000 7fc2327fc640 1 -- 172.21.15.60:0/3545036982 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc234031b40 con 0x7fc23c15f300 2024-05-09T05:01:49.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.410+0000 7fc2327fc640 1 -- 172.21.15.60:0/3545036982 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc234009450 con 0x7fc23c15f300 2024-05-09T05:01:49.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.410+0000 7fc243f40640 1 -- 172.21.15.60:0/3545036982 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc23c1f5960 con 0x7fc23c15f300 2024-05-09T05:01:49.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.410+0000 7fc243f40640 1 -- 172.21.15.60:0/3545036982 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc23c14d230 con 0x7fc23c15f300 2024-05-09T05:01:49.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.411+0000 7fc243f40640 1 -- 172.21.15.60:0/3545036982 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc23c051040 con 0x7fc23c15f300 2024-05-09T05:01:49.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.411+0000 7fc2327fc640 1 -- 172.21.15.60:0/3545036982 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc234039040 con 0x7fc23c15f300 2024-05-09T05:01:49.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.412+0000 7fc2327fc640 1 --2- 172.21.15.60:0/3545036982 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc2200762d0 0x7fc220078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.412+0000 7fc2327fc640 1 -- 172.21.15.60:0/3545036982 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6506+0+0 (secure 0 0 0) 0x7fc2340b8ca0 con 0x7fc23c15f300 2024-05-09T05:01:49.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.412+0000 7fc241cdc640 1 --2- 172.21.15.60:0/3545036982 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc2200762d0 0x7fc220078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.414 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.413+0000 7fc241cdc640 1 --2- 172.21.15.60:0/3545036982 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc2200762d0 0x7fc220078790 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc238000f60 tx=0x7fc238009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:49.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.416+0000 7fc2327fc640 1 -- 172.21.15.60:0/3545036982 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc234082370 con 0x7fc23c15f300 2024-05-09T05:01:49.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.610+0000 7fc243f40640 1 -- 172.21.15.60:0/3545036982 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc23c06f1b0 con 0x7fc23c15f300 2024-05-09T05:01:49.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.611+0000 7fc2327fc640 1 -- 172.21.15.60:0/3545036982 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) ==== 74+0+18747 (secure 0 0 0) 0x7fc234088020 con 0x7fc23c15f300 2024-05-09T05:01:49.612 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:49.612 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":53,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:01:48.278415+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:01:36.910324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T05:01:38.185514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data_extra1","create_time":"2024-05-09T05:01:47.082451+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:01:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.614+0000 7fc243f40640 1 -- 172.21.15.60:0/3545036982 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc2200762d0 msgr2=0x7fc220078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.614+0000 7fc243f40640 1 --2- 172.21.15.60:0/3545036982 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc2200762d0 0x7fc220078790 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc238000f60 tx=0x7fc238009040 comp rx=0 tx=0).stop 2024-05-09T05:01:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.614+0000 7fc243f40640 1 -- 172.21.15.60:0/3545036982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c15f300 msgr2=0x7fc23c0d3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.614+0000 7fc243f40640 1 --2- 172.21.15.60:0/3545036982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c15f300 0x7fc23c0d3940 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc234009d90 tx=0x7fc234004a40 comp rx=0 tx=0).stop 2024-05-09T05:01:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.614+0000 7fc243f40640 1 -- 172.21.15.60:0/3545036982 shutdown_connections 2024-05-09T05:01:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.614+0000 7fc243f40640 1 --2- 172.21.15.60:0/3545036982 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc2200762d0 0x7fc220078790 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.614+0000 7fc243f40640 1 --2- 172.21.15.60:0/3545036982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c15f300 0x7fc23c0d3940 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.614+0000 7fc243f40640 1 --2- 172.21.15.60:0/3545036982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc23c073de0 0x7fc23c1f4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.614+0000 7fc243f40640 1 --2- 172.21.15.60:0/3545036982 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc23c06fe20 0x7fc23c099fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.614+0000 7fc243f40640 1 -- 172.21.15.60:0/3545036982 >> 172.21.15.60:0/3545036982 conn(0x7fc23c0b4ef0 msgr2=0x7fc23c15cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:49.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.614+0000 7fc243f40640 1 -- 172.21.15.60:0/3545036982 shutdown_connections 2024-05-09T05:01:49.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.614+0000 7fc243f40640 1 -- 172.21.15.60:0/3545036982 wait complete. 2024-05-09T05:01:49.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:49.626 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_x) 2024-05-09T05:01:49.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:49.647 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.layout.pool -v cephfs_data_extra1 dir_x) 2024-05-09T05:01:49.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:49.707 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir_x/file_x) 2024-05-09T05:01:49.765 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 flush journal 2024-05-09T05:01:49.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.911+0000 7fb982558640 1 Processor -- start 2024-05-09T05:01:49.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.911+0000 7fb982558640 1 -- start start 2024-05-09T05:01:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.912+0000 7fb982558640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb97c160160 0x7fb97c160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.912+0000 7fb982558640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb97c0b84e0 0x7fb97c0ba230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.912+0000 7fb982558640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb97c0b8ef0 con 0x7fb97c0b8920 2024-05-09T05:01:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.912+0000 7fb982558640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fb97c0b9790 con 0x7fb97c0b84e0 2024-05-09T05:01:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.912+0000 7fb982558640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb97c0b91d0 con 0x7fb97c160160 2024-05-09T05:01:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.912+0000 7fb97b7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb97c0b84e0 0x7fb97c0ba230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.912+0000 7fb97b7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fb97c0b84e0 0x7fb97c0ba230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.60:34204/0 (socket says 172.21.15.60:34204) 2024-05-09T05:01:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.912+0000 7fb980af5640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fb97c0b8920 0x7fb97c162fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.60:57822/0 (socket says 172.21.15.60:57822) 2024-05-09T05:01:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.912+0000 7fb97b7fe640 1 -- 172.21.15.60:0/2059699671 learned_addr learned my addr 172.21.15.60:0/2059699671 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.912+0000 7fb97bfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb97c160160 0x7fb97c160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.913+0000 7fb97bfff640 1 -- 172.21.15.60:0/2059699671 >> v2:172.21.15.114:3300/0 conn(0x7fb97c0b84e0 msgr2=0x7fb97c0ba230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.913+0000 7fb97bfff640 1 --2- 172.21.15.60:0/2059699671 >> v2:172.21.15.114:3300/0 conn(0x7fb97c0b84e0 0x7fb97c0ba230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.913+0000 7fb97bfff640 1 -- 172.21.15.60:0/2059699671 >> v1:172.21.15.114:6789/0 conn(0x7fb97c0b8920 legacy=0x7fb97c162fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.913+0000 7fb97bfff640 1 -- 172.21.15.60:0/2059699671 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb97c0b94b0 con 0x7fb97c160160 2024-05-09T05:01:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.913+0000 7fb97b7fe640 1 --2- 172.21.15.60:0/2059699671 >> v2:172.21.15.114:3300/0 conn(0x7fb97c0b84e0 0x7fb97c0ba230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:01:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.913+0000 7fb97bfff640 1 --2- 172.21.15.60:0/2059699671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb97c160160 0x7fb97c160540 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb96c0097a0 tx=0x7fb96c02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a2d931ee376c70f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.915+0000 7fb97affd640 1 -- 172.21.15.60:0/2059699671 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96c037070 con 0x7fb97c160160 2024-05-09T05:01:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.915+0000 7fb97affd640 1 -- 172.21.15.60:0/2059699671 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb96c02ae50 con 0x7fb97c160160 2024-05-09T05:01:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.915+0000 7fb97affd640 1 -- 172.21.15.60:0/2059699671 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96c032660 con 0x7fb97c160160 2024-05-09T05:01:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.915+0000 7fb982558640 1 -- 172.21.15.60:0/2059699671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb97c160160 msgr2=0x7fb97c160540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.915+0000 7fb982558640 1 --2- 172.21.15.60:0/2059699671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb97c160160 0x7fb97c160540 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb96c0097a0 tx=0x7fb96c02a100 comp rx=0 tx=0).stop 2024-05-09T05:01:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.915+0000 7fb982558640 1 -- 172.21.15.60:0/2059699671 shutdown_connections 2024-05-09T05:01:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.915+0000 7fb982558640 1 --2- 172.21.15.60:0/2059699671 >> v2:172.21.15.114:3300/0 conn(0x7fb97c0b84e0 0x7fb97c0ba230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.915+0000 7fb982558640 1 --2- 172.21.15.60:0/2059699671 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb97c160160 0x7fb97c160540 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.915+0000 7fb982558640 1 -- 172.21.15.60:0/2059699671 >> 172.21.15.60:0/2059699671 conn(0x7fb97c06eda0 msgr2=0x7fb97c073820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:49.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.916+0000 7fb982558640 1 -- 172.21.15.60:0/2059699671 shutdown_connections 2024-05-09T05:01:49.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.916+0000 7fb982558640 1 -- 172.21.15.60:0/2059699671 wait complete. 2024-05-09T05:01:49.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.917+0000 7fb982558640 1 Processor -- start 2024-05-09T05:01:49.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.918+0000 7fb982558640 1 -- start start 2024-05-09T05:01:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.918+0000 7fb982558640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb97c0b84e0 0x7fb97c14bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.918+0000 7fb982558640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb97c0b8920 0x7fb97c0b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.918+0000 7fb982558640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb97c160160 0x7fb97c0b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.918+0000 7fb982558640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb97c15fd90 con 0x7fb97c160160 2024-05-09T05:01:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.918+0000 7fb982558640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb97c15fc10 con 0x7fb97c0b8920 2024-05-09T05:01:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.918+0000 7fb982558640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb97c0b6b10 con 0x7fb97c0b84e0 2024-05-09T05:01:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.918+0000 7fb980af5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb97c160160 0x7fb97c0b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.919+0000 7fb980af5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb97c160160 0x7fb97c0b3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35564/0 (socket says 172.21.15.60:35564) 2024-05-09T05:01:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.919+0000 7fb980af5640 1 -- 172.21.15.60:0/519093362 learned_addr learned my addr 172.21.15.60:0/519093362 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.919+0000 7fb97bfff640 1 --2- 172.21.15.60:0/519093362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb97c0b84e0 0x7fb97c14bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.919+0000 7fb97b7fe640 1 --2- 172.21.15.60:0/519093362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb97c0b8920 0x7fb97c0b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.919+0000 7fb980af5640 1 -- 172.21.15.60:0/519093362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb97c0b84e0 msgr2=0x7fb97c14bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.919+0000 7fb980af5640 1 --2- 172.21.15.60:0/519093362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb97c0b84e0 0x7fb97c14bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.919+0000 7fb980af5640 1 -- 172.21.15.60:0/519093362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb97c0b8920 msgr2=0x7fb97c0b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.919+0000 7fb980af5640 1 --2- 172.21.15.60:0/519093362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb97c0b8920 0x7fb97c0b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.919+0000 7fb980af5640 1 -- 172.21.15.60:0/519093362 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb97c0b1300 con 0x7fb97c160160 2024-05-09T05:01:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.919+0000 7fb980af5640 1 --2- 172.21.15.60:0/519093362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb97c160160 0x7fb97c0b3f70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fb97400c7f0 tx=0x7fb97400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.919+0000 7fb978ff9640 1 -- 172.21.15.60:0/519093362 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb974017070 con 0x7fb97c160160 2024-05-09T05:01:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.919+0000 7fb978ff9640 1 -- 172.21.15.60:0/519093362 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb97400a640 con 0x7fb97c160160 2024-05-09T05:01:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.919+0000 7fb982558640 1 -- 172.21.15.60:0/519093362 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb97c0b15e0 con 0x7fb97c160160 2024-05-09T05:01:49.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.919+0000 7fb978ff9640 1 -- 172.21.15.60:0/519093362 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97400a9f0 con 0x7fb97c160160 2024-05-09T05:01:49.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.919+0000 7fb982558640 1 -- 172.21.15.60:0/519093362 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb97c0b1ba0 con 0x7fb97c160160 2024-05-09T05:01:49.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.921+0000 7fb978ff9640 1 -- 172.21.15.60:0/519093362 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb97401a070 con 0x7fb97c160160 2024-05-09T05:01:49.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.921+0000 7fb978ff9640 1 --2- 172.21.15.60:0/519093362 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb9680762d0 0x7fb968078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.922+0000 7fb978ff9640 1 -- 172.21.15.60:0/519093362 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6506+0+0 (secure 0 0 0) 0x7fb974096e90 con 0x7fb97c160160 2024-05-09T05:01:49.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.922+0000 7fb97bfff640 1 --2- 172.21.15.60:0/519093362 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb9680762d0 0x7fb968078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.922+0000 7fb985273740 1 Processor -- start 2024-05-09T05:01:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.922+0000 7fb985273740 1 -- start start 2024-05-09T05:01:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.922+0000 7fb97bfff640 1 --2- 172.21.15.60:0/519093362 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb9680762d0 0x7fb968078790 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb96c034000 tx=0x7fb96c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.923+0000 7fb985273740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1c6200 0x56189a178b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.923+0000 7fb985273740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1a9320 0x56189a1a86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.923+0000 7fb985273740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1a8be0 0x56189a17d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.923+0000 7fb985273740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56189a145730 con 0x56189a1a9320 2024-05-09T05:01:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.923+0000 7fb985273740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56189a0e2f30 con 0x56189a1a8be0 2024-05-09T05:01:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.923+0000 7fb985273740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x56189a09bb20 con 0x56189a1c6200 2024-05-09T05:01:49.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.923+0000 7fb97b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1a9320 0x56189a1a86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.923+0000 7fb97bfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1c6200 0x56189a178b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.923+0000 7fb97bfff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1c6200 0x56189a178b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.60:34792/0 (socket says 172.21.15.60:34792) 2024-05-09T05:01:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.923+0000 7fb97b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1a9320 0x56189a1a86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35576/0 (socket says 172.21.15.60:35576) 2024-05-09T05:01:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.923+0000 7fb97bfff640 1 -- 172.21.15.60:0/4134730134 learned_addr learned my addr 172.21.15.60:0/4134730134 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.923+0000 7fb980af5640 1 --2- 172.21.15.60:0/4134730134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1a8be0 0x56189a17d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.923+0000 7fb97bfff640 1 -- 172.21.15.60:0/4134730134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1a8be0 msgr2=0x56189a17d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.923+0000 7fb97bfff640 1 --2- 172.21.15.60:0/4134730134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1a8be0 0x56189a17d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.923+0000 7fb97bfff640 1 -- 172.21.15.60:0/4134730134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1a9320 msgr2=0x56189a1a86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.923+0000 7fb97bfff640 1 --2- 172.21.15.60:0/4134730134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1a9320 0x56189a1a86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.923+0000 7fb97bfff640 1 -- 172.21.15.60:0/4134730134 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56189a093e60 con 0x56189a1c6200 2024-05-09T05:01:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.924+0000 7fb97bfff640 1 --2- 172.21.15.60:0/4134730134 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1c6200 0x56189a178b90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb96c009890 tx=0x7fb96c02c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32d6513fdd87f3f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.924+0000 7fb961ffb640 1 -- 172.21.15.60:0/4134730134 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96c037040 con 0x56189a1c6200 2024-05-09T05:01:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.924+0000 7fb961ffb640 1 -- 172.21.15.60:0/4134730134 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb96c04c070 con 0x56189a1c6200 2024-05-09T05:01:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.924+0000 7fb961ffb640 1 -- 172.21.15.60:0/4134730134 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96c032df0 con 0x56189a1c6200 2024-05-09T05:01:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.924+0000 7fb985273740 1 -- 172.21.15.60:0/4134730134 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1c6200 msgr2=0x56189a178b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.924+0000 7fb985273740 1 --2- 172.21.15.60:0/4134730134 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1c6200 0x56189a178b90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb96c009890 tx=0x7fb96c02c000 comp rx=0 tx=0).stop 2024-05-09T05:01:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.924+0000 7fb985273740 1 -- 172.21.15.60:0/4134730134 shutdown_connections 2024-05-09T05:01:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.924+0000 7fb985273740 1 --2- 172.21.15.60:0/4134730134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1a8be0 0x56189a17d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.924+0000 7fb985273740 1 --2- 172.21.15.60:0/4134730134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1a9320 0x56189a1a86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.924+0000 7fb985273740 1 --2- 172.21.15.60:0/4134730134 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1c6200 0x56189a178b90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.924+0000 7fb985273740 1 -- 172.21.15.60:0/4134730134 >> 172.21.15.60:0/4134730134 conn(0x56189a17ae00 msgr2=0x56189a1c36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:49.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.924+0000 7fb985273740 1 -- 172.21.15.60:0/4134730134 shutdown_connections 2024-05-09T05:01:49.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.925+0000 7fb985273740 1 -- 172.21.15.60:0/4134730134 wait complete. 2024-05-09T05:01:49.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.925+0000 7fb985273740 1 -- start start 2024-05-09T05:01:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.925+0000 7fb985273740 1 Processor -- start 2024-05-09T05:01:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.926+0000 7fb985273740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1a9320 0x56189a195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.926+0000 7fb985273740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 0x56189a1abe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.926+0000 7fb985273740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1966d0 0x56189a1a98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.926+0000 7fb985273740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56189a09f5e0 con 0x56189a1c6200 2024-05-09T05:01:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.926+0000 7fb985273740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56189a0deef0 con 0x56189a1966d0 2024-05-09T05:01:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.926+0000 7fb985273740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x56189a17cfb0 con 0x56189a1a9320 2024-05-09T05:01:49.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.926+0000 7fb985273740 10 client.0 _ll_register_callbacks cb 0x56189a09a320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:01:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.926+0000 7fb97b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 0x56189a1abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.926+0000 7fb97b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 0x56189a1abe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35578/0 (socket says 172.21.15.60:35578) 2024-05-09T05:01:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.926+0000 7fb97b7fe640 1 -- 172.21.15.60:0/2149411575 learned_addr learned my addr 172.21.15.60:0/2149411575 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.926+0000 7fb97bfff640 1 --2- 172.21.15.60:0/2149411575 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1a9320 0x56189a195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.926+0000 7fb980af5640 1 --2- 172.21.15.60:0/2149411575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1966d0 0x56189a1a98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.926+0000 7fb97b7fe640 1 -- 172.21.15.60:0/2149411575 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1a9320 msgr2=0x56189a195fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.926+0000 7fb97b7fe640 1 --2- 172.21.15.60:0/2149411575 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1a9320 0x56189a195fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.926+0000 7fb97b7fe640 1 -- 172.21.15.60:0/2149411575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1966d0 msgr2=0x56189a1a98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.926+0000 7fb97b7fe640 1 --2- 172.21.15.60:0/2149411575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1966d0 0x56189a1a98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.926+0000 7fb97b7fe640 1 -- 172.21.15.60:0/2149411575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561899ebe7d0 con 0x56189a1c6200 2024-05-09T05:01:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.926+0000 7fb980af5640 1 --2- 172.21.15.60:0/2149411575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1966d0 0x56189a1a98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.927+0000 7fb97b7fe640 1 --2- 172.21.15.60:0/2149411575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 0x56189a1abe00 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fb9700147d0 tx=0x7fb970014c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfc9e50dc09531f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:49.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.927+0000 7fb95affd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-05-09T05:01:49.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.927+0000 7fb95affd640 1 -- 172.21.15.60:0/2149411575 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb970012b20 con 0x56189a1c6200 2024-05-09T05:01:49.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.927+0000 7fb95affd640 1 -- 172.21.15.60:0/2149411575 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(54..54 src has 1..54) ==== 6506+0+0 (secure 0 0 0) 0x7fb970012cc0 con 0x56189a1c6200 2024-05-09T05:01:49.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.927+0000 7fb985273740 1 -- 172.21.15.60:0/2149411575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb96c04c070 con 0x56189a1c6200 2024-05-09T05:01:49.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.927+0000 7fb95affd640 1 -- 172.21.15.60:0/2149411575 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb970004020 con 0x56189a1c6200 2024-05-09T05:01:49.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.927+0000 7fb985273740 10 client.5586 fetch_fsmap learned FSMap version 28 2024-05-09T05:01:49.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.927+0000 7fb985273740 1 -- 172.21.15.60:0/2149411575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) -- 0x561899df43f0 con 0x56189a1c6200 2024-05-09T05:01:49.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.927+0000 7fb95affd640 1 -- 172.21.15.60:0/2149411575 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb96c04c070 con 0x56189a1c6200 2024-05-09T05:01:49.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.927+0000 7fb95affd640 1 -- 172.21.15.60:0/2149411575 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) ==== 3502+0+0 (secure 0 0 0) 0x7fb97002a080 con 0x56189a1c6200 2024-05-09T05:01:49.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.927+0000 7fb985273740 10 client.5586 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T05:01:49.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.927+0000 7fb985273740 10 client.5586 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-05-09T05:01:49.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.927+0000 7fb985273740 20 client.5586 populate_metadata read hostname 'smithi060' 2024-05-09T05:01:49.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.927+0000 7fb985273740 1 --2- 172.21.15.60:0/2149411575 >> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] conn(0x56189a196b50 0x56189a1c4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.927+0000 7fb985273740 4 client.5586 mds_command: new command op to 5415 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T05:01:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.927+0000 7fb985273740 1 -- 172.21.15.60:0/2149411575 --> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56189a0f0b40 con 0x56189a196b50 2024-05-09T05:01:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.927+0000 7fb97bfff640 1 --2- 172.21.15.60:0/2149411575 >> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] conn(0x56189a196b50 0x56189a1c4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.928+0000 7fb95affd640 10 client.5586 ms_handle_connect on v2:172.21.15.60:6833/2425263408 2024-05-09T05:01:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.928+0000 7fb97bfff640 1 --2- 172.21.15.60:0/2149411575 >> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] conn(0x56189a196b50 0x56189a1c4a60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7486f782bfdfd97 server_cookie=3caa0fd50f4395be in_seq=0 out_seq=0 2024-05-09T05:01:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.928+0000 7fb95affd640 1 -- 172.21.15.60:0/2149411575 <== mds.0 v2:172.21.15.60:6833/2425263408 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56189a0f0b40 con 0x56189a196b50 2024-05-09T05:01:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.928+0000 7fb95affd640 10 client.5586 handle_command_reply: tid=0 2024-05-09T05:01:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.928+0000 7fb985273740 1 client.5586 shutdown 2024-05-09T05:01:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.929+0000 7fb985273740 1 -- 172.21.15.60:0/2149411575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 msgr2=0x56189a1abe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.929+0000 7fb985273740 1 --2- 172.21.15.60:0/2149411575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 0x56189a1abe00 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fb9700147d0 tx=0x7fb970014c90 comp rx=0 tx=0).stop 2024-05-09T05:01:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.929+0000 7fb985273740 1 -- 172.21.15.60:0/2149411575 shutdown_connections 2024-05-09T05:01:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.929+0000 7fb985273740 1 --2- 172.21.15.60:0/2149411575 >> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] conn(0x56189a196b50 0x56189a1c4a60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.929+0000 7fb985273740 1 --2- 172.21.15.60:0/2149411575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1966d0 0x56189a1a98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.929+0000 7fb985273740 1 --2- 172.21.15.60:0/2149411575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 0x56189a1abe00 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.929+0000 7fb95affd640 0 client.5586 ms_handle_reset on v2:172.21.15.60:6833/2425263408 2024-05-09T05:01:49.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.929+0000 7fb985273740 1 --2- 172.21.15.60:0/2149411575 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1a9320 0x56189a195fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.929+0000 7fb985273740 1 -- 172.21.15.60:0/2149411575 >> 172.21.15.60:0/2149411575 conn(0x56189a1a8be0 msgr2=0x56189a1782d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:49.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.929+0000 7fb985273740 1 -- 172.21.15.60:0/2149411575 shutdown_connections 2024-05-09T05:01:49.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.929+0000 7fb985273740 1 -- 172.21.15.60:0/2149411575 wait complete. 2024-05-09T05:01:49.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.929+0000 7fb985273740 20 client.5586 trim_cache size 0 max 16384 2024-05-09T05:01:49.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.944+0000 7fb985273740 1 Processor -- start 2024-05-09T05:01:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.944+0000 7fb985273740 1 -- start start 2024-05-09T05:01:49.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb985273740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1966d0 0x56189a1abde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb985273740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1a9320 0x56189a1a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb985273740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 0x56189a1a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb985273740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb97c202cc0 con 0x56189a1c6200 2024-05-09T05:01:49.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb985273740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb97c201cc0 con 0x56189a1966d0 2024-05-09T05:01:49.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb985273740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb97c201020 con 0x56189a1a9320 2024-05-09T05:01:49.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb980af5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 0x56189a1a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb980af5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 0x56189a1a36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35588/0 (socket says 172.21.15.60:35588) 2024-05-09T05:01:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb980af5640 1 -- 172.21.15.60:0/2842256877 learned_addr learned my addr 172.21.15.60:0/2842256877 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb980af5640 1 -- 172.21.15.60:0/2842256877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1a9320 msgr2=0x56189a1a4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb97bfff640 1 --2- 172.21.15.60:0/2842256877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1966d0 0x56189a1abde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb97b7fe640 1 --2- 172.21.15.60:0/2842256877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1a9320 0x56189a1a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb980af5640 1 --2- 172.21.15.60:0/2842256877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1a9320 0x56189a1a4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb980af5640 1 -- 172.21.15.60:0/2842256877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1966d0 msgr2=0x56189a1abde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb980af5640 1 --2- 172.21.15.60:0/2842256877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1966d0 0x56189a1abde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb980af5640 1 -- 172.21.15.60:0/2842256877 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561899db40f0 con 0x56189a1c6200 2024-05-09T05:01:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb980af5640 1 --2- 172.21.15.60:0/2842256877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 0x56189a1a36e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb97400c7c0 tx=0x7fb97405f990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0141f0db1d2d2f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb9617fa640 1 -- 172.21.15.60:0/2842256877 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb974063040 con 0x56189a1c6200 2024-05-09T05:01:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb9617fa640 1 -- 172.21.15.60:0/2842256877 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb974064180 con 0x56189a1c6200 2024-05-09T05:01:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.945+0000 7fb9617fa640 1 -- 172.21.15.60:0/2842256877 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb974068910 con 0x56189a1c6200 2024-05-09T05:01:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.946+0000 7fb985273740 1 -- 172.21.15.60:0/2842256877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 msgr2=0x56189a1a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.946+0000 7fb985273740 1 --2- 172.21.15.60:0/2842256877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 0x56189a1a36e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb97400c7c0 tx=0x7fb97405f990 comp rx=0 tx=0).stop 2024-05-09T05:01:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.946+0000 7fb985273740 1 -- 172.21.15.60:0/2842256877 shutdown_connections 2024-05-09T05:01:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.946+0000 7fb985273740 1 --2- 172.21.15.60:0/2842256877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 0x56189a1a36e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.946+0000 7fb985273740 1 --2- 172.21.15.60:0/2842256877 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1a9320 0x56189a1a4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.946+0000 7fb985273740 1 --2- 172.21.15.60:0/2842256877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1966d0 0x56189a1abde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.946+0000 7fb985273740 1 -- 172.21.15.60:0/2842256877 >> 172.21.15.60:0/2842256877 conn(0x56189a1a8be0 msgr2=0x56189a1c34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.946+0000 7fb985273740 1 -- 172.21.15.60:0/2842256877 shutdown_connections 2024-05-09T05:01:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.946+0000 7fb985273740 1 -- 172.21.15.60:0/2842256877 wait complete. 2024-05-09T05:01:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.947+0000 7fb985273740 1 -- start start 2024-05-09T05:01:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.947+0000 7fb985273740 1 Processor -- start 2024-05-09T05:01:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb985273740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1966d0 0x56189a240bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb985273740 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1a9320 0x56189a1a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb985273740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 0x56189a1a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb985273740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x56189a17fa10 con 0x56189a1c6200 2024-05-09T05:01:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb985273740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb97c201670 con 0x56189a1966d0 2024-05-09T05:01:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb985273740 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x56189a1e7460 con 0x56189a1a9320 2024-05-09T05:01:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb985273740 10 client.0 _ll_register_callbacks cb 0x7fb9700097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:01:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb980af5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 0x56189a1a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb980af5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 0x56189a1a53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35598/0 (socket says 172.21.15.60:35598) 2024-05-09T05:01:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb980af5640 1 -- 172.21.15.60:0/2709676299 learned_addr learned my addr 172.21.15.60:0/2709676299 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb97bfff640 1 --2- 172.21.15.60:0/2709676299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1966d0 0x56189a240bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb97b7fe640 1 --2- 172.21.15.60:0/2709676299 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1a9320 0x56189a1a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb980af5640 1 -- 172.21.15.60:0/2709676299 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1a9320 msgr2=0x56189a1a4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb980af5640 1 --2- 172.21.15.60:0/2709676299 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1a9320 0x56189a1a4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb980af5640 1 -- 172.21.15.60:0/2709676299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1966d0 msgr2=0x56189a240bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb980af5640 1 --2- 172.21.15.60:0/2709676299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1966d0 0x56189a240bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb980af5640 1 -- 172.21.15.60:0/2709676299 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56189a1c6f50 con 0x56189a1c6200 2024-05-09T05:01:49.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb980af5640 1 --2- 172.21.15.60:0/2709676299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 0x56189a1a53d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb97405fdc0 tx=0x7fb974007050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b09a998e2946cf63 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb95affd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-05-09T05:01:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb95affd640 1 -- 172.21.15.60:0/2709676299 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97407d020 con 0x56189a1c6200 2024-05-09T05:01:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb95affd640 1 -- 172.21.15.60:0/2709676299 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(54..54 src has 1..54) ==== 6506+0+0 (secure 0 0 0) 0x7fb974068910 con 0x56189a1c6200 2024-05-09T05:01:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.948+0000 7fb985273740 1 -- 172.21.15.60:0/2709676299 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb974064180 con 0x56189a1c6200 2024-05-09T05:01:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.949+0000 7fb95affd640 1 -- 172.21.15.60:0/2709676299 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb9740813a0 con 0x56189a1c6200 2024-05-09T05:01:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.949+0000 7fb985273740 10 client.5592 fetch_fsmap learned FSMap version 28 2024-05-09T05:01:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.949+0000 7fb985273740 1 -- 172.21.15.60:0/2709676299 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) -- 0x56189a1791c0 con 0x56189a1c6200 2024-05-09T05:01:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.949+0000 7fb95affd640 1 -- 172.21.15.60:0/2709676299 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb974064180 con 0x56189a1c6200 2024-05-09T05:01:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.949+0000 7fb95affd640 1 -- 172.21.15.60:0/2709676299 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) ==== 3502+0+0 (secure 0 0 0) 0x7fb97400ab90 con 0x56189a1c6200 2024-05-09T05:01:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.949+0000 7fb985273740 10 client.5592 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T05:01:49.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.949+0000 7fb985273740 10 client.5592 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-05-09T05:01:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.949+0000 7fb985273740 20 client.5592 populate_metadata read hostname 'smithi060' 2024-05-09T05:01:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.949+0000 7fb985273740 1 --2- 172.21.15.60:0/2709676299 >> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] conn(0x56189a1c50a0 0x56189a1b9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.949+0000 7fb985273740 4 client.5592 mds_command: new command op to 5415 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T05:01:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.949+0000 7fb985273740 1 -- 172.21.15.60:0/2709676299 --> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56189a0ea9b0 con 0x56189a1c50a0 2024-05-09T05:01:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.949+0000 7fb97bfff640 1 --2- 172.21.15.60:0/2709676299 >> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] conn(0x56189a1c50a0 0x56189a1b9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.949+0000 7fb95affd640 10 client.5592 ms_handle_connect on v2:172.21.15.60:6833/2425263408 2024-05-09T05:01:49.957 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.949+0000 7fb97bfff640 1 --2- 172.21.15.60:0/2709676299 >> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] conn(0x56189a1c50a0 0x56189a1b9030 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d41810091f2ca7a5 server_cookie=5ef7a3beac617a39 in_seq=0 out_seq=0 2024-05-09T05:01:49.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.960+0000 7fb95affd640 1 -- 172.21.15.60:0/2709676299 <== mds.0 v2:172.21.15.60:6833/2425263408 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56189a0ea9b0 con 0x56189a1c50a0 2024-05-09T05:01:49.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.960+0000 7fb95affd640 10 client.5592 handle_command_reply: tid=0 2024-05-09T05:01:49.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.960+0000 7fb985273740 1 client.5592 shutdown 2024-05-09T05:01:49.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.961+0000 7fb985273740 1 -- 172.21.15.60:0/2709676299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 msgr2=0x56189a1a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.961+0000 7fb985273740 1 --2- 172.21.15.60:0/2709676299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 0x56189a1a53d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb97405fdc0 tx=0x7fb974007050 comp rx=0 tx=0).stop 2024-05-09T05:01:49.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.961+0000 7fb985273740 1 -- 172.21.15.60:0/2709676299 shutdown_connections 2024-05-09T05:01:49.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.961+0000 7fb985273740 1 --2- 172.21.15.60:0/2709676299 >> [v2:172.21.15.60:6833/2425263408,v1:172.21.15.60:6835/2425263408] conn(0x56189a1c50a0 0x56189a1b9030 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.961+0000 7fb985273740 1 --2- 172.21.15.60:0/2709676299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56189a1c6200 0x56189a1a53d0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.961+0000 7fb985273740 1 --2- 172.21.15.60:0/2709676299 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x56189a1a9320 0x56189a1a4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.961+0000 7fb95affd640 0 client.5592 ms_handle_reset on v2:172.21.15.60:6833/2425263408 2024-05-09T05:01:49.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.961+0000 7fb985273740 1 --2- 172.21.15.60:0/2709676299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56189a1966d0 0x56189a240bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.961+0000 7fb985273740 1 -- 172.21.15.60:0/2709676299 >> 172.21.15.60:0/2709676299 conn(0x56189a1a8be0 msgr2=0x56189a174050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:49.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.961+0000 7fb985273740 1 -- 172.21.15.60:0/2709676299 shutdown_connections 2024-05-09T05:01:49.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.961+0000 7fb985273740 1 -- 172.21.15.60:0/2709676299 wait complete. 2024-05-09T05:01:49.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.961+0000 7fb985273740 20 client.5592 trim_cache size 0 max 16384 2024-05-09T05:01:49.963 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:01:49.963 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "", 2024-05-09T05:01:49.963 INFO:teuthology.orchestra.run.smithi060.stdout: "return_code": 0 2024-05-09T05:01:49.963 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:01:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.961+0000 7fb982558640 1 -- 172.21.15.60:0/519093362 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb9680762d0 msgr2=0x7fb968078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.961+0000 7fb982558640 1 --2- 172.21.15.60:0/519093362 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb9680762d0 0x7fb968078790 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb96c034000 tx=0x7fb96c02e000 comp rx=0 tx=0).stop 2024-05-09T05:01:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.961+0000 7fb982558640 1 -- 172.21.15.60:0/519093362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb97c160160 msgr2=0x7fb97c0b3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:49.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.961+0000 7fb982558640 1 --2- 172.21.15.60:0/519093362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb97c160160 0x7fb97c0b3f70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fb97400c7f0 tx=0x7fb97400ccb0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.962+0000 7fb982558640 1 -- 172.21.15.60:0/519093362 shutdown_connections 2024-05-09T05:01:49.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.962+0000 7fb982558640 1 --2- 172.21.15.60:0/519093362 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb9680762d0 0x7fb968078790 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.962+0000 7fb982558640 1 --2- 172.21.15.60:0/519093362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb97c160160 0x7fb97c0b3f70 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.962+0000 7fb982558640 1 --2- 172.21.15.60:0/519093362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb97c0b8920 0x7fb97c0b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.962+0000 7fb982558640 1 --2- 172.21.15.60:0/519093362 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb97c0b84e0 0x7fb97c14bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:49.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.962+0000 7fb982558640 1 -- 172.21.15.60:0/519093362 >> 172.21.15.60:0/519093362 conn(0x7fb97c06eda0 msgr2=0x7fb97c1608e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:49.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.962+0000 7fb982558640 1 -- 172.21.15.60:0/519093362 shutdown_connections 2024-05-09T05:01:49.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:49.962+0000 7fb982558640 1 -- 172.21.15.60:0/519093362 wait complete. 2024-05-09T05:01:49.973 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_extra2 --pg_num_min 64 2024-05-09T05:01:50.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.082+0000 7f8e85f04640 1 Processor -- start 2024-05-09T05:01:50.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.082+0000 7f8e85f04640 1 -- start start 2024-05-09T05:01:50.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.083+0000 7f8e85f04640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e801600f0 0x7f8e8016a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:50.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.083+0000 7f8e85f04640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e8015d250 con 0x7f8e8015ce70 2024-05-09T05:01:50.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.083+0000 7f8e85f04640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e80160730 con 0x7f8e800b4770 2024-05-09T05:01:50.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.083+0000 7f8e85f04640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8e8006c370 con 0x7f8e801600f0 2024-05-09T05:01:50.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.083+0000 7f8e7f7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8e8015ce70 0x7f8e800b4060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57128/0 (socket says 172.21.15.60:57128) 2024-05-09T05:01:50.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.083+0000 7f8e7f7fe640 1 -- 172.21.15.60:0/1690053204 learned_addr learned my addr 172.21.15.60:0/1690053204 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:50.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.083+0000 7f8e7ffff640 1 --2- 172.21.15.60:0/1690053204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e801600f0 0x7f8e8016a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:50.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.083+0000 7f8e7e7fc640 1 -- 172.21.15.60:0/1690053204 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2393123650 0 0) 0x7f8e8015d250 con 0x7f8e8015ce70 2024-05-09T05:01:50.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.083+0000 7f8e7e7fc640 1 -- 172.21.15.60:0/1690053204 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e68003520 con 0x7f8e8015ce70 2024-05-09T05:01:50.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.083+0000 7f8e7e7fc640 1 -- 172.21.15.60:0/1690053204 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1716640757 0 0) 0x7f8e68003520 con 0x7f8e8015ce70 2024-05-09T05:01:50.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.083+0000 7f8e7e7fc640 1 -- 172.21.15.60:0/1690053204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e801600f0 msgr2=0x7f8e8016a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:50.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.083+0000 7f8e7e7fc640 1 --2- 172.21.15.60:0/1690053204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e801600f0 0x7f8e8016a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:50.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.083+0000 7f8e7e7fc640 1 -- 172.21.15.60:0/1690053204 >> v1:172.21.15.114:6789/0 conn(0x7f8e800b4770 legacy=0x7f8e8015f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:50.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.083+0000 7f8e7e7fc640 1 -- 172.21.15.60:0/1690053204 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e8006c650 con 0x7f8e8015ce70 2024-05-09T05:01:50.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.083+0000 7f8e7e7fc640 1 -- 172.21.15.60:0/1690053204 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f8e700016f0 con 0x7f8e8015ce70 2024-05-09T05:01:50.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.083+0000 7f8e7e7fc640 1 -- 172.21.15.60:0/1690053204 >> v1:172.21.15.60:6789/0 conn(0x7f8e8015ce70 legacy=0x7f8e800b4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:50.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.084+0000 7f8e7e7fc640 1 --2- 172.21.15.60:0/1690053204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e68003a30 0x7f8e68003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:50.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.084+0000 7f8e7e7fc640 1 --2- 172.21.15.60:0/1690053204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e68004800 0x7f8e68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:50.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.084+0000 7f8e7e7fc640 1 --2- 172.21.15.60:0/1690053204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e801600f0 0x7f8e68005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:50.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.084+0000 7f8e7e7fc640 1 -- 172.21.15.60:0/1690053204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8006c370 con 0x7f8e68004800 2024-05-09T05:01:50.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.084+0000 7f8e7e7fc640 1 -- 172.21.15.60:0/1690053204 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8e6800bb80 con 0x7f8e68003a30 2024-05-09T05:01:50.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.084+0000 7f8e7e7fc640 1 -- 172.21.15.60:0/1690053204 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8e6800bd80 con 0x7f8e801600f0 2024-05-09T05:01:50.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.084+0000 7f8e7effd640 1 --2- 172.21.15.60:0/1690053204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e68004800 0x7f8e68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:50.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.084+0000 7f8e7f7fe640 1 --2- 172.21.15.60:0/1690053204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e68003a30 0x7f8e68003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:50.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.084+0000 7f8e7ffff640 1 --2- 172.21.15.60:0/1690053204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e801600f0 0x7f8e68005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:50.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.084+0000 7f8e7effd640 1 -- 172.21.15.60:0/1690053204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e801600f0 msgr2=0x7f8e68005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:50.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.084+0000 7f8e7effd640 1 --2- 172.21.15.60:0/1690053204 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e801600f0 0x7f8e68005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:50.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.084+0000 7f8e7effd640 1 -- 172.21.15.60:0/1690053204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e68003a30 msgr2=0x7f8e68003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:50.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.084+0000 7f8e7effd640 1 --2- 172.21.15.60:0/1690053204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e68003a30 0x7f8e68003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:50.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.084+0000 7f8e7effd640 1 -- 172.21.15.60:0/1690053204 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e68003520 con 0x7f8e68004800 2024-05-09T05:01:50.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.085+0000 7f8e7effd640 1 --2- 172.21.15.60:0/1690053204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e68004800 0x7f8e68004c80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8e74001790 tx=0x7f8e68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef45719fb33dceb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:50.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.085+0000 7f8e7e7fc640 1 -- 172.21.15.60:0/1690053204 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e740013b0 con 0x7f8e68004800 2024-05-09T05:01:50.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.085+0000 7f8e7e7fc640 1 -- 172.21.15.60:0/1690053204 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e740052d0 con 0x7f8e68004800 2024-05-09T05:01:50.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.085+0000 7f8e7e7fc640 1 -- 172.21.15.60:0/1690053204 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7402aae0 con 0x7f8e68004800 2024-05-09T05:01:50.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.085+0000 7f8e85f04640 1 -- 172.21.15.60:0/1690053204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e68004800 msgr2=0x7f8e68004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:50.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.085+0000 7f8e85f04640 1 --2- 172.21.15.60:0/1690053204 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e68004800 0x7f8e68004c80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8e74001790 tx=0x7f8e68005830 comp rx=0 tx=0).stop 2024-05-09T05:01:50.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.085+0000 7f8e7ffff640 1 -- 172.21.15.60:0/1690053204 reap_dead start 2024-05-09T05:01:50.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.085+0000 7f8e85f04640 1 -- 172.21.15.60:0/1690053204 shutdown_connections 2024-05-09T05:01:50.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.085+0000 7f8e85f04640 1 -- 172.21.15.60:0/1690053204 >> 172.21.15.60:0/1690053204 conn(0x7f8e800b3c80 msgr2=0x7f8e80070590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:50.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.085+0000 7f8e85f04640 1 -- 172.21.15.60:0/1690053204 shutdown_connections 2024-05-09T05:01:50.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.085+0000 7f8e85f04640 1 -- 172.21.15.60:0/1690053204 wait complete. 2024-05-09T05:01:50.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.087+0000 7f8e85f04640 1 Processor -- start 2024-05-09T05:01:50.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.087+0000 7f8e85f04640 1 -- start start 2024-05-09T05:01:50.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.087+0000 7f8e85f04640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e68004800 0x7f8e8014c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:50.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.087+0000 7f8e85f04640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e8014fff0 0x7f8e80150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:50.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.087+0000 7f8e85f04640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e80161a80 0x7f8e80161f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:50.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.087+0000 7f8e85f04640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8006d2b0 con 0x7f8e68004800 2024-05-09T05:01:50.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.087+0000 7f8e85f04640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8006d130 con 0x7f8e80161a80 2024-05-09T05:01:50.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.087+0000 7f8e85f04640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8e80168270 con 0x7f8e8014fff0 2024-05-09T05:01:50.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.088+0000 7f8e7f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e68004800 0x7f8e8014c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:50.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.088+0000 7f8e7f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e68004800 0x7f8e8014c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35614/0 (socket says 172.21.15.60:35614) 2024-05-09T05:01:50.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.088+0000 7f8e7f7fe640 1 -- 172.21.15.60:0/2580279527 learned_addr learned my addr 172.21.15.60:0/2580279527 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:50.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.088+0000 7f8e7ffff640 1 --2- 172.21.15.60:0/2580279527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e80161a80 0x7f8e80161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:50.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.088+0000 7f8e7effd640 1 --2- 172.21.15.60:0/2580279527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e8014fff0 0x7f8e80150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:50.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.088+0000 7f8e7f7fe640 1 -- 172.21.15.60:0/2580279527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e8014fff0 msgr2=0x7f8e80150450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:50.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.088+0000 7f8e7f7fe640 1 --2- 172.21.15.60:0/2580279527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e8014fff0 0x7f8e80150450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:50.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.088+0000 7f8e7f7fe640 1 -- 172.21.15.60:0/2580279527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e80161a80 msgr2=0x7f8e80161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:50.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.088+0000 7f8e7f7fe640 1 --2- 172.21.15.60:0/2580279527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e80161a80 0x7f8e80161f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:50.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.088+0000 7f8e7f7fe640 1 -- 172.21.15.60:0/2580279527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e80157c20 con 0x7f8e68004800 2024-05-09T05:01:50.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.088+0000 7f8e7f7fe640 1 --2- 172.21.15.60:0/2580279527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e68004800 0x7f8e8014c7a0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8e70007290 tx=0x7f8e70007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:50.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.088+0000 7f8e67fff640 1 -- 172.21.15.60:0/2580279527 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e700016e0 con 0x7f8e68004800 2024-05-09T05:01:50.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.088+0000 7f8e67fff640 1 -- 172.21.15.60:0/2580279527 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e70018070 con 0x7f8e68004800 2024-05-09T05:01:50.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.088+0000 7f8e85f04640 1 -- 172.21.15.60:0/2580279527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e80157f00 con 0x7f8e68004800 2024-05-09T05:01:50.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.088+0000 7f8e67fff640 1 -- 172.21.15.60:0/2580279527 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7000fba0 con 0x7f8e68004800 2024-05-09T05:01:50.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.088+0000 7f8e85f04640 1 -- 172.21.15.60:0/2580279527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e801584c0 con 0x7f8e68004800 2024-05-09T05:01:50.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.089+0000 7f8e85f04640 1 -- 172.21.15.60:0/2580279527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e801586c0 con 0x7f8e68004800 2024-05-09T05:01:50.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.090+0000 7f8e67fff640 1 -- 172.21.15.60:0/2580279527 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8e7000fd40 con 0x7f8e68004800 2024-05-09T05:01:50.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.090+0000 7f8e67fff640 1 --2- 172.21.15.60:0/2580279527 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8e6c0762d0 0x7f8e6c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:50.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.090+0000 7f8e67fff640 1 -- 172.21.15.60:0/2580279527 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6506+0+0 (secure 0 0 0) 0x7f8e700976c0 con 0x7f8e68004800 2024-05-09T05:01:50.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.091+0000 7f8e7effd640 1 --2- 172.21.15.60:0/2580279527 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8e6c0762d0 0x7f8e6c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:50.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.091+0000 7f8e7effd640 1 --2- 172.21.15.60:0/2580279527 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8e6c0762d0 0x7f8e6c078790 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8e74032000 tx=0x7f8e7402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:50.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.095+0000 7f8e67fff640 1 -- 172.21.15.60:0/2580279527 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8e801586c0 con 0x7f8e68004800 2024-05-09T05:01:50.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.285+0000 7f8e85f04640 1 -- 172.21.15.60:0/2580279527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_extra2", "pg_num_min": 64} v 0) -- 0x7f8e8006d9d0 con 0x7f8e68004800 2024-05-09T05:01:50.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.921+0000 7f8e67fff640 1 -- 172.21.15.60:0/2580279527 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_extra2", "pg_num_min": 64}]=0 pool 'cephfs_data_extra2' created v55) ==== 144+0+0 (secure 0 0 0) 0x7f8e70060740 con 0x7f8e68004800 2024-05-09T05:01:50.921 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data_extra2' created 2024-05-09T05:01:50.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.925+0000 7f8e85f04640 1 -- 172.21.15.60:0/2580279527 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8e6c0762d0 msgr2=0x7f8e6c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:50.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.925+0000 7f8e85f04640 1 --2- 172.21.15.60:0/2580279527 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8e6c0762d0 0x7f8e6c078790 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8e74032000 tx=0x7f8e7402e000 comp rx=0 tx=0).stop 2024-05-09T05:01:50.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.925+0000 7f8e85f04640 1 -- 172.21.15.60:0/2580279527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e68004800 msgr2=0x7f8e8014c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:50.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.925+0000 7f8e85f04640 1 --2- 172.21.15.60:0/2580279527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e68004800 0x7f8e8014c7a0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8e70007290 tx=0x7f8e70007750 comp rx=0 tx=0).stop 2024-05-09T05:01:50.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.925+0000 7f8e85f04640 1 -- 172.21.15.60:0/2580279527 shutdown_connections 2024-05-09T05:01:50.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.925+0000 7f8e85f04640 1 --2- 172.21.15.60:0/2580279527 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8e6c0762d0 0x7f8e6c078790 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:50.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.925+0000 7f8e85f04640 1 --2- 172.21.15.60:0/2580279527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e80161a80 0x7f8e80161f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:50.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.925+0000 7f8e85f04640 1 --2- 172.21.15.60:0/2580279527 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8e8014fff0 0x7f8e80150450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:50.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.925+0000 7f8e85f04640 1 --2- 172.21.15.60:0/2580279527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8e68004800 0x7f8e8014c7a0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:50.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.925+0000 7f8e85f04640 1 -- 172.21.15.60:0/2580279527 >> 172.21.15.60:0/2580279527 conn(0x7f8e800b3c80 msgr2=0x7f8e8015d600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:50.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.926+0000 7f8e85f04640 1 -- 172.21.15.60:0/2580279527 shutdown_connections 2024-05-09T05:01:50.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:50.926+0000 7f8e85f04640 1 -- 172.21.15.60:0/2580279527 wait complete. 2024-05-09T05:01:50.945 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_extra2 2024-05-09T05:01:51.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.057+0000 7f76d6bf0640 1 Processor -- start 2024-05-09T05:01:51.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.057+0000 7f76d6bf0640 1 -- start start 2024-05-09T05:01:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.058+0000 7f76d6bf0640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76d00b6c40 0x7f76d00b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.058+0000 7f76d6bf0640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f76d00d1d50 0x7f76d00d21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.058+0000 7f76d6bf0640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76d00b55e0 con 0x7f76d015f0b0 2024-05-09T05:01:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.058+0000 7f76d6bf0640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f76d00b75e0 con 0x7f76d00d1d50 2024-05-09T05:01:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.058+0000 7f76d6bf0640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f76d0163d50 con 0x7f76d00b6c40 2024-05-09T05:01:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.058+0000 7f76d518d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f76d00d1d50 0x7f76d00d21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.058+0000 7f76cffff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76d00b6c40 0x7f76d00b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.058+0000 7f76d498c640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f76d015f0b0 0x7f76d00b63a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.60:57844/0 (socket says 172.21.15.60:57844) 2024-05-09T05:01:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.058+0000 7f76d498c640 1 -- 172.21.15.60:0/4103706751 learned_addr learned my addr 172.21.15.60:0/4103706751 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.058+0000 7f76d518d640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f76d00d1d50 0x7f76d00d21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.60:34284/0 (socket says 172.21.15.60:34284) 2024-05-09T05:01:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.058+0000 7f76cf7fe640 1 -- 172.21.15.60:0/4103706751 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2121182065 0 0) 0x7f76d00b55e0 con 0x7f76d015f0b0 2024-05-09T05:01:51.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.058+0000 7f76cf7fe640 1 -- 172.21.15.60:0/4103706751 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76c0003520 con 0x7f76d015f0b0 2024-05-09T05:01:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.058+0000 7f76d518d640 1 -- 172.21.15.60:0/4103706751 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76d00b6c40 msgr2=0x7f76d00b70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.058+0000 7f76d518d640 1 --2- 172.21.15.60:0/4103706751 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76d00b6c40 0x7f76d00b70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.058+0000 7f76d518d640 1 -- 172.21.15.60:0/4103706751 >> v1:172.21.15.114:6789/0 conn(0x7f76d015f0b0 legacy=0x7f76d00b63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.058+0000 7f76d518d640 1 -- 172.21.15.60:0/4103706751 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d00b58c0 con 0x7f76d00d1d50 2024-05-09T05:01:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.058+0000 7f76cffff640 1 --2- 172.21.15.60:0/4103706751 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76d00b6c40 0x7f76d00b70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:01:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.059+0000 7f76d518d640 1 --2- 172.21.15.60:0/4103706751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d00d1d50 0x7f76d00d21a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009d90 tx=0x7f76c802a530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9827cb30587cf523 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.059+0000 7f76cf7fe640 1 -- 172.21.15.60:0/4103706751 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c8007740 con 0x7f76d00d1d50 2024-05-09T05:01:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.059+0000 7f76cf7fe640 1 -- 172.21.15.60:0/4103706751 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76c80078e0 con 0x7f76d00d1d50 2024-05-09T05:01:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.059+0000 7f76cf7fe640 1 -- 172.21.15.60:0/4103706751 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c8007c90 con 0x7f76d00d1d50 2024-05-09T05:01:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.060+0000 7f76d6bf0640 1 -- 172.21.15.60:0/4103706751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d00d1d50 msgr2=0x7f76d00d21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.060+0000 7f76d6bf0640 1 --2- 172.21.15.60:0/4103706751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d00d1d50 0x7f76d00d21a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009d90 tx=0x7f76c802a530 comp rx=0 tx=0).stop 2024-05-09T05:01:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.060+0000 7f76d6bf0640 1 -- 172.21.15.60:0/4103706751 shutdown_connections 2024-05-09T05:01:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.060+0000 7f76d6bf0640 1 --2- 172.21.15.60:0/4103706751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d00d1d50 0x7f76d00d21a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.060+0000 7f76d6bf0640 1 --2- 172.21.15.60:0/4103706751 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76d00b6c40 0x7f76d00b70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.060+0000 7f76d6bf0640 1 -- 172.21.15.60:0/4103706751 >> 172.21.15.60:0/4103706751 conn(0x7f76d006ed30 msgr2=0x7f76d0073820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:51.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.060+0000 7f76d6bf0640 1 -- 172.21.15.60:0/4103706751 shutdown_connections 2024-05-09T05:01:51.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.060+0000 7f76d6bf0640 1 -- 172.21.15.60:0/4103706751 wait complete. 2024-05-09T05:01:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.061+0000 7f76d6bf0640 1 Processor -- start 2024-05-09T05:01:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.061+0000 7f76d6bf0640 1 -- start start 2024-05-09T05:01:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.062+0000 7f76d6bf0640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76d00b6c40 0x7f76d014dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.062+0000 7f76d6bf0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d00d1d50 0x7f76d01541c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.062+0000 7f76d6bf0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d015f0b0 0x7f76d01607e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.062+0000 7f76d6bf0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f76d00d26e0 con 0x7f76d015f0b0 2024-05-09T05:01:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.062+0000 7f76d6bf0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f76d00b5e80 con 0x7f76d00d1d50 2024-05-09T05:01:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.062+0000 7f76d6bf0640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f76d01604a0 con 0x7f76d00b6c40 2024-05-09T05:01:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.062+0000 7f76d518d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d015f0b0 0x7f76d01607e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.062+0000 7f76d518d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d015f0b0 0x7f76d01607e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35616/0 (socket says 172.21.15.60:35616) 2024-05-09T05:01:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.062+0000 7f76d518d640 1 -- 172.21.15.60:0/1109115232 learned_addr learned my addr 172.21.15.60:0/1109115232 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.062+0000 7f76d498c640 1 --2- 172.21.15.60:0/1109115232 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76d00b6c40 0x7f76d014dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:51.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.062+0000 7f76cffff640 1 --2- 172.21.15.60:0/1109115232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d00d1d50 0x7f76d01541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:51.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.063+0000 7f76d518d640 1 -- 172.21.15.60:0/1109115232 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76d00b6c40 msgr2=0x7f76d014dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:51.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.063+0000 7f76d518d640 1 --2- 172.21.15.60:0/1109115232 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76d00b6c40 0x7f76d014dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:51.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.063+0000 7f76d518d640 1 -- 172.21.15.60:0/1109115232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d00d1d50 msgr2=0x7f76d01541c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:51.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.063+0000 7f76d518d640 1 --2- 172.21.15.60:0/1109115232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d00d1d50 0x7f76d01541c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:51.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.063+0000 7f76d518d640 1 -- 172.21.15.60:0/1109115232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d01527c0 con 0x7f76d015f0b0 2024-05-09T05:01:51.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.063+0000 7f76d518d640 1 --2- 172.21.15.60:0/1109115232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d015f0b0 0x7f76d01607e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009d90 tx=0x7f76c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:51.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.063+0000 7f76cd7fa640 1 -- 172.21.15.60:0/1109115232 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c8040070 con 0x7f76d015f0b0 2024-05-09T05:01:51.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.063+0000 7f76cd7fa640 1 -- 172.21.15.60:0/1109115232 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76c8031c40 con 0x7f76d015f0b0 2024-05-09T05:01:51.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.063+0000 7f76d6bf0640 1 -- 172.21.15.60:0/1109115232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d0152aa0 con 0x7f76d015f0b0 2024-05-09T05:01:51.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.063+0000 7f76cd7fa640 1 -- 172.21.15.60:0/1109115232 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c8009450 con 0x7f76d015f0b0 2024-05-09T05:01:51.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.063+0000 7f76d6bf0640 1 -- 172.21.15.60:0/1109115232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d014e0e0 con 0x7f76d015f0b0 2024-05-09T05:01:51.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.064+0000 7f76d6bf0640 1 -- 172.21.15.60:0/1109115232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76d0051040 con 0x7f76d015f0b0 2024-05-09T05:01:51.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.065+0000 7f76cd7fa640 1 -- 172.21.15.60:0/1109115232 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f76c8039040 con 0x7f76d015f0b0 2024-05-09T05:01:51.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.066+0000 7f76cd7fa640 1 --2- 172.21.15.60:0/1109115232 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f76b40762d0 0x7f76b4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:51.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.066+0000 7f76cd7fa640 1 -- 172.21.15.60:0/1109115232 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6879+0+0 (secure 0 0 0) 0x7f76c80b8ab0 con 0x7f76d015f0b0 2024-05-09T05:01:51.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.066+0000 7f76d498c640 1 --2- 172.21.15.60:0/1109115232 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f76b40762d0 0x7f76b4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:51.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.067+0000 7f76d498c640 1 --2- 172.21.15.60:0/1109115232 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f76b40762d0 0x7f76b4078790 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f76c4005880 tx=0x7f76c4006aa0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:51.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.071+0000 7f76cd7fa640 1 -- 172.21.15.60:0/1109115232 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f76c8081920 con 0x7f76d015f0b0 2024-05-09T05:01:51.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.303+0000 7f76d6bf0640 1 -- 172.21.15.60:0/1109115232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_extra2"} v 0) -- 0x7f76d00d1b30 con 0x7f76d015f0b0 2024-05-09T05:01:51.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.922+0000 7f76cd7fa640 1 -- 172.21.15.60:0/1109115232 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_extra2"}]=0 added data pool 11 to fsmap v29) ==== 142+0+0 (secure 0 0 0) 0x7f76c8085c20 con 0x7f76d015f0b0 2024-05-09T05:01:51.923 INFO:teuthology.orchestra.run.smithi060.stderr:added data pool 11 to fsmap 2024-05-09T05:01:51.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.925+0000 7f76d6bf0640 1 -- 172.21.15.60:0/1109115232 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f76b40762d0 msgr2=0x7f76b4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:51.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.925+0000 7f76d6bf0640 1 --2- 172.21.15.60:0/1109115232 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f76b40762d0 0x7f76b4078790 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f76c4005880 tx=0x7f76c4006aa0 comp rx=0 tx=0).stop 2024-05-09T05:01:51.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.925+0000 7f76d6bf0640 1 -- 172.21.15.60:0/1109115232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d015f0b0 msgr2=0x7f76d01607e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:51.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.925+0000 7f76d6bf0640 1 --2- 172.21.15.60:0/1109115232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d015f0b0 0x7f76d01607e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009d90 tx=0x7f76c8004a40 comp rx=0 tx=0).stop 2024-05-09T05:01:51.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.927+0000 7f76d6bf0640 1 -- 172.21.15.60:0/1109115232 shutdown_connections 2024-05-09T05:01:51.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.927+0000 7f76d6bf0640 1 --2- 172.21.15.60:0/1109115232 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f76b40762d0 0x7f76b4078790 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:51.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.927+0000 7f76d6bf0640 1 --2- 172.21.15.60:0/1109115232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76d015f0b0 0x7f76d01607e0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:51.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.927+0000 7f76d6bf0640 1 --2- 172.21.15.60:0/1109115232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76d00d1d50 0x7f76d01541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:51.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.927+0000 7f76d6bf0640 1 --2- 172.21.15.60:0/1109115232 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f76d00b6c40 0x7f76d014dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:51.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.927+0000 7f76d6bf0640 1 -- 172.21.15.60:0/1109115232 >> 172.21.15.60:0/1109115232 conn(0x7f76d006ed30 msgr2=0x7f76d00b9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:51.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.928+0000 7f76d6bf0640 1 -- 172.21.15.60:0/1109115232 shutdown_connections 2024-05-09T05:01:51.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:51.929+0000 7f76d6bf0640 1 -- 172.21.15.60:0/1109115232 wait complete. 2024-05-09T05:01:51.953 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:52.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.062+0000 7fc5fca75640 1 Processor -- start 2024-05-09T05:01:52.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.062+0000 7fc5fca75640 1 -- start start 2024-05-09T05:01:52.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.063+0000 7fc5fca75640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5f8096d70 0x7fc5f809d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:52.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.063+0000 7fc5fca75640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f8097150 con 0x7fc5f80b2580 2024-05-09T05:01:52.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.063+0000 7fc5fca75640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f809c020 con 0x7fc5f80b1040 2024-05-09T05:01:52.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.063+0000 7fc5fca75640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc5f809c300 con 0x7fc5f8096d70 2024-05-09T05:01:52.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.063+0000 7fc5f6d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc5f80b2580 0x7fc5f80b2990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57144/0 (socket says 172.21.15.60:57144) 2024-05-09T05:01:52.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.063+0000 7fc5f6d9d640 1 -- 172.21.15.60:0/1303217429 learned_addr learned my addr 172.21.15.60:0/1303217429 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:52.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.063+0000 7fc5f5d9b640 1 -- 172.21.15.60:0/1303217429 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3184445693 0 0) 0x7fc5f8097150 con 0x7fc5f80b2580 2024-05-09T05:01:52.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.063+0000 7fc5f659c640 1 --2- 172.21.15.60:0/1303217429 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5f8096d70 0x7fc5f809d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:52.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.063+0000 7fc5f5d9b640 1 -- 172.21.15.60:0/1303217429 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5e4003520 con 0x7fc5f80b2580 2024-05-09T05:01:52.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.063+0000 7fc5f5d9b640 1 -- 172.21.15.60:0/1303217429 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4157774047 0 0) 0x7fc5e4003520 con 0x7fc5f80b2580 2024-05-09T05:01:52.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.063+0000 7fc5f5d9b640 1 -- 172.21.15.60:0/1303217429 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5f8096d70 msgr2=0x7fc5f809d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:52.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.063+0000 7fc5f5d9b640 1 --2- 172.21.15.60:0/1303217429 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5f8096d70 0x7fc5f809d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:52.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.063+0000 7fc5f5d9b640 1 -- 172.21.15.60:0/1303217429 >> v1:172.21.15.114:6789/0 conn(0x7fc5f80b1040 legacy=0x7fc5f80b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:52.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.064+0000 7fc5f5d9b640 1 -- 172.21.15.60:0/1303217429 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f809c5e0 con 0x7fc5f80b2580 2024-05-09T05:01:52.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.064+0000 7fc5f5d9b640 1 -- 172.21.15.60:0/1303217429 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7fc5f0001a70 con 0x7fc5f80b2580 2024-05-09T05:01:52.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.064+0000 7fc5f5d9b640 1 -- 172.21.15.60:0/1303217429 >> v1:172.21.15.60:6789/0 conn(0x7fc5f80b2580 legacy=0x7fc5f80b2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:52.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.064+0000 7fc5f5d9b640 1 --2- 172.21.15.60:0/1303217429 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5e4003a30 0x7fc5e4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:52.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.064+0000 7fc5f5d9b640 1 --2- 172.21.15.60:0/1303217429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5e4004800 0x7fc5e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:52.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.064+0000 7fc5f5d9b640 1 --2- 172.21.15.60:0/1303217429 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5f8096d70 0x7fc5e4005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:52.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.064+0000 7fc5f5d9b640 1 -- 172.21.15.60:0/1303217429 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f809c300 con 0x7fc5e4004800 2024-05-09T05:01:52.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.064+0000 7fc5f5d9b640 1 -- 172.21.15.60:0/1303217429 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e400bb80 con 0x7fc5e4003a30 2024-05-09T05:01:52.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.064+0000 7fc5f5d9b640 1 -- 172.21.15.60:0/1303217429 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc5e400bd80 con 0x7fc5f8096d70 2024-05-09T05:01:52.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.064+0000 7fc5effff640 1 --2- 172.21.15.60:0/1303217429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5e4004800 0x7fc5e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:52.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.064+0000 7fc5f659c640 1 --2- 172.21.15.60:0/1303217429 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5e4003a30 0x7fc5e4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:52.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.064+0000 7fc5f6d9d640 1 --2- 172.21.15.60:0/1303217429 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5f8096d70 0x7fc5e4005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:52.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.064+0000 7fc5effff640 1 -- 172.21.15.60:0/1303217429 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5f8096d70 msgr2=0x7fc5e4005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:52.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.064+0000 7fc5effff640 1 --2- 172.21.15.60:0/1303217429 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5f8096d70 0x7fc5e4005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:52.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.064+0000 7fc5effff640 1 -- 172.21.15.60:0/1303217429 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5e4003a30 msgr2=0x7fc5e4003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:52.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.064+0000 7fc5effff640 1 --2- 172.21.15.60:0/1303217429 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5e4003a30 0x7fc5e4003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:52.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.064+0000 7fc5effff640 1 -- 172.21.15.60:0/1303217429 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e4003520 con 0x7fc5e4004800 2024-05-09T05:01:52.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.064+0000 7fc5f6d9d640 1 --2- 172.21.15.60:0/1303217429 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5f8096d70 0x7fc5e4005290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:52.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.065+0000 7fc5effff640 1 --2- 172.21.15.60:0/1303217429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5e4004800 0x7fc5e4004c80 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0001730 tx=0x7fc5e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a81cfb6170dd37 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:52.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.065+0000 7fc5f5d9b640 1 -- 172.21.15.60:0/1303217429 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e0001350 con 0x7fc5e4004800 2024-05-09T05:01:52.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.065+0000 7fc5f5d9b640 1 -- 172.21.15.60:0/1303217429 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5e00052d0 con 0x7fc5e4004800 2024-05-09T05:01:52.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.065+0000 7fc5f5d9b640 1 -- 172.21.15.60:0/1303217429 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e002aac0 con 0x7fc5e4004800 2024-05-09T05:01:52.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.065+0000 7fc5fca75640 1 -- 172.21.15.60:0/1303217429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5e4004800 msgr2=0x7fc5e4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:52.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.065+0000 7fc5fca75640 1 --2- 172.21.15.60:0/1303217429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5e4004800 0x7fc5e4004c80 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0001730 tx=0x7fc5e4005830 comp rx=0 tx=0).stop 2024-05-09T05:01:52.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.065+0000 7fc5f6d9d640 1 -- 172.21.15.60:0/1303217429 reap_dead start 2024-05-09T05:01:52.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.065+0000 7fc5fca75640 1 -- 172.21.15.60:0/1303217429 shutdown_connections 2024-05-09T05:01:52.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.065+0000 7fc5fca75640 1 -- 172.21.15.60:0/1303217429 >> 172.21.15.60:0/1303217429 conn(0x7fc5f8072390 msgr2=0x7fc5f80727a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:52.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.066+0000 7fc5fca75640 1 -- 172.21.15.60:0/1303217429 shutdown_connections 2024-05-09T05:01:52.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.066+0000 7fc5fca75640 1 -- 172.21.15.60:0/1303217429 wait complete. 2024-05-09T05:01:52.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.067+0000 7fc5fca75640 1 Processor -- start 2024-05-09T05:01:52.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.067+0000 7fc5fca75640 1 -- start start 2024-05-09T05:01:52.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.068+0000 7fc5fca75640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5e4004800 0x7fc5f8158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:52.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.068+0000 7fc5fca75640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5f806fc40 0x7fc5f81f4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:52.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.068+0000 7fc5fca75640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5f81f53d0 0x7fc5f81f5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:52.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.068+0000 7fc5fca75640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f809bc40 con 0x7fc5f806fc40 2024-05-09T05:01:52.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.068+0000 7fc5fca75640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f809cba0 con 0x7fc5e4004800 2024-05-09T05:01:52.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.068+0000 7fc5fca75640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc5f809bdc0 con 0x7fc5f81f53d0 2024-05-09T05:01:52.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.068+0000 7fc5effff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5f806fc40 0x7fc5f81f4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:52.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.068+0000 7fc5effff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5f806fc40 0x7fc5f81f4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35640/0 (socket says 172.21.15.60:35640) 2024-05-09T05:01:52.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.068+0000 7fc5effff640 1 -- 172.21.15.60:0/1661453594 learned_addr learned my addr 172.21.15.60:0/1661453594 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:52.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.068+0000 7fc5f659c640 1 --2- 172.21.15.60:0/1661453594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5e4004800 0x7fc5f8158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:52.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.068+0000 7fc5f6d9d640 1 --2- 172.21.15.60:0/1661453594 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5f81f53d0 0x7fc5f81f5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:52.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.068+0000 7fc5effff640 1 -- 172.21.15.60:0/1661453594 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5f81f53d0 msgr2=0x7fc5f81f5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:52.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.068+0000 7fc5effff640 1 --2- 172.21.15.60:0/1661453594 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5f81f53d0 0x7fc5f81f5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:52.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.068+0000 7fc5effff640 1 -- 172.21.15.60:0/1661453594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5e4004800 msgr2=0x7fc5f8158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:52.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.068+0000 7fc5effff640 1 --2- 172.21.15.60:0/1661453594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5e4004800 0x7fc5f8158310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:52.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.068+0000 7fc5effff640 1 -- 172.21.15.60:0/1661453594 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f8158a10 con 0x7fc5f806fc40 2024-05-09T05:01:52.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.068+0000 7fc5effff640 1 --2- 172.21.15.60:0/1661453594 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5f806fc40 0x7fc5f81f4e90 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0001850 tx=0x7fc5e0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:52.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.069+0000 7fc5eeffd640 1 -- 172.21.15.60:0/1661453594 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e0040020 con 0x7fc5f806fc40 2024-05-09T05:01:52.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.069+0000 7fc5eeffd640 1 -- 172.21.15.60:0/1661453594 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5e0005480 con 0x7fc5f806fc40 2024-05-09T05:01:52.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.069+0000 7fc5fca75640 1 -- 172.21.15.60:0/1661453594 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5f8158cf0 con 0x7fc5f806fc40 2024-05-09T05:01:52.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.069+0000 7fc5eeffd640 1 -- 172.21.15.60:0/1661453594 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e00069e0 con 0x7fc5f806fc40 2024-05-09T05:01:52.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.069+0000 7fc5fca75640 1 -- 172.21.15.60:0/1661453594 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5f81592b0 con 0x7fc5f806fc40 2024-05-09T05:01:52.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.070+0000 7fc5fca75640 1 -- 172.21.15.60:0/1661453594 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5f80b3c30 con 0x7fc5f806fc40 2024-05-09T05:01:52.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.070+0000 7fc5eeffd640 1 -- 172.21.15.60:0/1661453594 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc5e0043020 con 0x7fc5f806fc40 2024-05-09T05:01:52.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.071+0000 7fc5eeffd640 1 --2- 172.21.15.60:0/1661453594 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc5d80762d0 0x7fc5d8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:52.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.071+0000 7fc5eeffd640 1 -- 172.21.15.60:0/1661453594 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6911+0+0 (secure 0 0 0) 0x7fc5e00b79d0 con 0x7fc5f806fc40 2024-05-09T05:01:52.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.071+0000 7fc5f659c640 1 --2- 172.21.15.60:0/1661453594 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc5d80762d0 0x7fc5d8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:52.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.072+0000 7fc5f659c640 1 --2- 172.21.15.60:0/1661453594 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc5d80762d0 0x7fc5d8078790 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8009bc0 tx=0x7fc5e8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:52.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.075+0000 7fc5eeffd640 1 -- 172.21.15.60:0/1661453594 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc5f80b3c30 con 0x7fc5f806fc40 2024-05-09T05:01:52.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.305+0000 7fc5fca75640 1 -- 172.21.15.60:0/1661453594 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5f8051040 con 0x7fc5f806fc40 2024-05-09T05:01:52.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.306+0000 7fc5eeffd640 1 -- 172.21.15.60:0/1661453594 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+5193 (secure 0 0 0) 0x7fc5e004a020 con 0x7fc5f806fc40 2024-05-09T05:01:52.307 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:52.307 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":29,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3604145570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3604145570},{"type":"v1","addr":"172.21.15.60:6834","nonce":3604145570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3949194455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3949194455},{"type":"v1","addr":"172.21.15.114:6836","nonce":3949194455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3288630382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3288630382},{"type":"v1","addr":"172.21.15.114:6837","nonce":3288630382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:39.237498+0000","modified":"2024-05-09T05:01:51.305520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5415":{"gid":5415,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2425263408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2425263408},{"type":"v1","addr":"172.21.15.60:6835","nonce":2425263408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9,10,11],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5415,"qdb_cluster":[5415]},"id":3}]} 2024-05-09T05:01:52.307 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 29 2024-05-09T05:01:52.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.308+0000 7fc5fca75640 1 -- 172.21.15.60:0/1661453594 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc5d80762d0 msgr2=0x7fc5d8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:52.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.308+0000 7fc5fca75640 1 --2- 172.21.15.60:0/1661453594 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc5d80762d0 0x7fc5d8078790 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8009bc0 tx=0x7fc5e8009190 comp rx=0 tx=0).stop 2024-05-09T05:01:52.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.308+0000 7fc5fca75640 1 -- 172.21.15.60:0/1661453594 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5f806fc40 msgr2=0x7fc5f81f4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:52.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.308+0000 7fc5fca75640 1 --2- 172.21.15.60:0/1661453594 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5f806fc40 0x7fc5f81f4e90 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0001850 tx=0x7fc5e0001120 comp rx=0 tx=0).stop 2024-05-09T05:01:52.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.308+0000 7fc5fca75640 1 -- 172.21.15.60:0/1661453594 shutdown_connections 2024-05-09T05:01:52.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.308+0000 7fc5fca75640 1 --2- 172.21.15.60:0/1661453594 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc5d80762d0 0x7fc5d8078790 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:52.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.308+0000 7fc5fca75640 1 --2- 172.21.15.60:0/1661453594 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc5f81f53d0 0x7fc5f81f5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:52.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.308+0000 7fc5fca75640 1 --2- 172.21.15.60:0/1661453594 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5f806fc40 0x7fc5f81f4e90 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:52.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.308+0000 7fc5fca75640 1 --2- 172.21.15.60:0/1661453594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5e4004800 0x7fc5f8158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:52.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.308+0000 7fc5fca75640 1 -- 172.21.15.60:0/1661453594 >> 172.21.15.60:0/1661453594 conn(0x7fc5f8072390 msgr2=0x7fc5f80980d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:52.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.308+0000 7fc5fca75640 1 -- 172.21.15.60:0/1661453594 shutdown_connections 2024-05-09T05:01:52.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.308+0000 7fc5fca75640 1 -- 172.21.15.60:0/1661453594 wait complete. 2024-05-09T05:01:52.320 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:01:52.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.423+0000 7fdc6fada640 1 Processor -- start 2024-05-09T05:01:52.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.423+0000 7fdc6fada640 1 -- start start 2024-05-09T05:01:52.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.424+0000 7fdc6fada640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fdc681606f0 0x7fdc680d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:52.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.424+0000 7fdc6fada640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdc6806f9f0 0x7fdc680d3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:52.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.424+0000 7fdc6fada640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc6806ff60 con 0x7fdc6815f300 2024-05-09T05:01:52.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.424+0000 7fdc6fada640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fdc68070240 con 0x7fdc6806f9f0 2024-05-09T05:01:52.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.424+0000 7fdc6fada640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fdc68070520 con 0x7fdc681606f0 2024-05-09T05:01:52.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.424+0000 7fdc6e077640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdc6806f9f0 0x7fdc680d3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:52.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.424+0000 7fdc6e077640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdc6806f9f0 0x7fdc680d3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35654/0 (socket says 172.21.15.60:35654) 2024-05-09T05:01:52.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.424+0000 7fdc6e077640 1 -- 172.21.15.60:0/2468938646 learned_addr learned my addr 172.21.15.60:0/2468938646 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:52.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.424+0000 7fdc6d075640 1 --2- 172.21.15.60:0/2468938646 >> v2:172.21.15.114:3300/0 conn(0x7fdc681606f0 0x7fdc680d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:52.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.424+0000 7fdc6e077640 1 -- 172.21.15.60:0/2468938646 >> v2:172.21.15.114:3300/0 conn(0x7fdc681606f0 msgr2=0x7fdc680d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:52.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.424+0000 7fdc6e077640 1 --2- 172.21.15.60:0/2468938646 >> v2:172.21.15.114:3300/0 conn(0x7fdc681606f0 0x7fdc680d33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:52.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.424+0000 7fdc6e077640 1 -- 172.21.15.60:0/2468938646 >> v1:172.21.15.114:6789/0 conn(0x7fdc6815f300 legacy=0x7fdc681639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:52.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.424+0000 7fdc6e077640 1 -- 172.21.15.60:0/2468938646 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc6806e740 con 0x7fdc6806f9f0 2024-05-09T05:01:52.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.425+0000 7fdc6e077640 1 --2- 172.21.15.60:0/2468938646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc6806f9f0 0x7fdc680d3930 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fdc60009e50 tx=0x7fdc6002a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f87e41716fffd056 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:52.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.425+0000 7fdc6c874640 1 -- 172.21.15.60:0/2468938646 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc60036070 con 0x7fdc6806f9f0 2024-05-09T05:01:52.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.425+0000 7fdc6c874640 1 -- 172.21.15.60:0/2468938646 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc60031600 con 0x7fdc6806f9f0 2024-05-09T05:01:52.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.425+0000 7fdc6c874640 1 -- 172.21.15.60:0/2468938646 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc600319d0 con 0x7fdc6806f9f0 2024-05-09T05:01:52.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.426+0000 7fdc6fada640 1 -- 172.21.15.60:0/2468938646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc6806f9f0 msgr2=0x7fdc680d3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:52.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.426+0000 7fdc6fada640 1 --2- 172.21.15.60:0/2468938646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc6806f9f0 0x7fdc680d3930 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fdc60009e50 tx=0x7fdc6002a760 comp rx=0 tx=0).stop 2024-05-09T05:01:52.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.426+0000 7fdc6fada640 1 -- 172.21.15.60:0/2468938646 shutdown_connections 2024-05-09T05:01:52.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.426+0000 7fdc6fada640 1 --2- 172.21.15.60:0/2468938646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc6806f9f0 0x7fdc680d3930 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:52.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.426+0000 7fdc6fada640 1 --2- 172.21.15.60:0/2468938646 >> v2:172.21.15.114:3300/0 conn(0x7fdc681606f0 0x7fdc680d33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:52.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.426+0000 7fdc6fada640 1 -- 172.21.15.60:0/2468938646 >> 172.21.15.60:0/2468938646 conn(0x7fdc6809b850 msgr2=0x7fdc6809bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:52.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.426+0000 7fdc6fada640 1 -- 172.21.15.60:0/2468938646 shutdown_connections 2024-05-09T05:01:52.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.426+0000 7fdc6fada640 1 -- 172.21.15.60:0/2468938646 wait complete. 2024-05-09T05:01:52.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.427+0000 7fdc6fada640 1 Processor -- start 2024-05-09T05:01:52.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.427+0000 7fdc6fada640 1 -- start start 2024-05-09T05:01:52.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.428+0000 7fdc6fada640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc6806f9f0 0x7fdc681f6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:52.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.428+0000 7fdc6fada640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc6815f300 0x7fdc681611c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:52.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.428+0000 7fdc6fada640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc681606f0 0x7fdc68150080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:52.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.428+0000 7fdc6fada640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6806f0c0 con 0x7fdc6806f9f0 2024-05-09T05:01:52.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.428+0000 7fdc6fada640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6806ef40 con 0x7fdc681606f0 2024-05-09T05:01:52.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.428+0000 7fdc6fada640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fdc6806f240 con 0x7fdc6815f300 2024-05-09T05:01:52.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.428+0000 7fdc6d876640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc6806f9f0 0x7fdc681f6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:52.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.428+0000 7fdc6d876640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc6806f9f0 0x7fdc681f6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35664/0 (socket says 172.21.15.60:35664) 2024-05-09T05:01:52.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.428+0000 7fdc6d876640 1 -- 172.21.15.60:0/1196077854 learned_addr learned my addr 172.21.15.60:0/1196077854 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:52.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.428+0000 7fdc6d075640 1 --2- 172.21.15.60:0/1196077854 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc6815f300 0x7fdc681611c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:52.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.428+0000 7fdc6e077640 1 --2- 172.21.15.60:0/1196077854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc681606f0 0x7fdc68150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:52.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.428+0000 7fdc6d876640 1 -- 172.21.15.60:0/1196077854 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc6815f300 msgr2=0x7fdc681611c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:52.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.428+0000 7fdc6d876640 1 --2- 172.21.15.60:0/1196077854 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc6815f300 0x7fdc681611c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:52.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.428+0000 7fdc6d876640 1 -- 172.21.15.60:0/1196077854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc681606f0 msgr2=0x7fdc68150080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:52.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.428+0000 7fdc6d876640 1 --2- 172.21.15.60:0/1196077854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc681606f0 0x7fdc68150080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:52.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.428+0000 7fdc6d876640 1 -- 172.21.15.60:0/1196077854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc681f9400 con 0x7fdc6806f9f0 2024-05-09T05:01:52.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.428+0000 7fdc6d876640 1 --2- 172.21.15.60:0/1196077854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc6806f9f0 0x7fdc681f6460 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fdc6400c450 tx=0x7fdc6400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:52.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.429+0000 7fdc5e7fc640 1 -- 172.21.15.60:0/1196077854 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc640052d0 con 0x7fdc6806f9f0 2024-05-09T05:01:52.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.429+0000 7fdc5e7fc640 1 -- 172.21.15.60:0/1196077854 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc64005470 con 0x7fdc6806f9f0 2024-05-09T05:01:52.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.429+0000 7fdc6fada640 1 -- 172.21.15.60:0/1196077854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc681f96e0 con 0x7fdc6806f9f0 2024-05-09T05:01:52.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.429+0000 7fdc5e7fc640 1 -- 172.21.15.60:0/1196077854 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc64012870 con 0x7fdc6806f9f0 2024-05-09T05:01:52.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.429+0000 7fdc6fada640 1 -- 172.21.15.60:0/1196077854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc681f9ca0 con 0x7fdc6806f9f0 2024-05-09T05:01:52.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.430+0000 7fdc6fada640 1 -- 172.21.15.60:0/1196077854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc680d0970 con 0x7fdc6806f9f0 2024-05-09T05:01:52.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.430+0000 7fdc5e7fc640 1 -- 172.21.15.60:0/1196077854 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdc64012a10 con 0x7fdc6806f9f0 2024-05-09T05:01:52.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.431+0000 7fdc5e7fc640 1 --2- 172.21.15.60:0/1196077854 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fdc4c076300 0x7fdc4c0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:52.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.431+0000 7fdc5e7fc640 1 -- 172.21.15.60:0/1196077854 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6911+0+0 (secure 0 0 0) 0x7fdc64097700 con 0x7fdc6806f9f0 2024-05-09T05:01:52.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.431+0000 7fdc6d075640 1 --2- 172.21.15.60:0/1196077854 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fdc4c076300 0x7fdc4c0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:52.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.431+0000 7fdc6d075640 1 --2- 172.21.15.60:0/1196077854 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fdc4c076300 0x7fdc4c0787c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdc6809f830 tx=0x7fdc58008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:52.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.435+0000 7fdc5e7fc640 1 -- 172.21.15.60:0/1196077854 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fdc680d0970 con 0x7fdc6806f9f0 2024-05-09T05:01:52.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.627+0000 7fdc6fada640 1 -- 172.21.15.60:0/1196077854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdc68051040 con 0x7fdc6806f9f0 2024-05-09T05:01:52.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.628+0000 7fdc5e7fc640 1 -- 172.21.15.60:0/1196077854 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+20570 (secure 0 0 0) 0x7fdc64060b30 con 0x7fdc6806f9f0 2024-05-09T05:01:52.629 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:52.629 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":56,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:01:51.305305+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:01:36.910324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T05:01:38.185514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data_extra1","create_time":"2024-05-09T05:01:47.082451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data_extra2","create_time":"2024-05-09T05:01:50.290433+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:01:52.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.630+0000 7fdc6fada640 1 -- 172.21.15.60:0/1196077854 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fdc4c076300 msgr2=0x7fdc4c0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:52.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.630+0000 7fdc6fada640 1 --2- 172.21.15.60:0/1196077854 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fdc4c076300 0x7fdc4c0787c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdc6809f830 tx=0x7fdc58008040 comp rx=0 tx=0).stop 2024-05-09T05:01:52.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.630+0000 7fdc6fada640 1 -- 172.21.15.60:0/1196077854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc6806f9f0 msgr2=0x7fdc681f6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:52.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.630+0000 7fdc6fada640 1 --2- 172.21.15.60:0/1196077854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc6806f9f0 0x7fdc681f6460 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fdc6400c450 tx=0x7fdc6400c910 comp rx=0 tx=0).stop 2024-05-09T05:01:52.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.630+0000 7fdc6fada640 1 -- 172.21.15.60:0/1196077854 shutdown_connections 2024-05-09T05:01:52.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.630+0000 7fdc6fada640 1 --2- 172.21.15.60:0/1196077854 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fdc4c076300 0x7fdc4c0787c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:52.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.630+0000 7fdc6fada640 1 --2- 172.21.15.60:0/1196077854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc681606f0 0x7fdc68150080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:52.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.630+0000 7fdc6fada640 1 --2- 172.21.15.60:0/1196077854 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fdc6815f300 0x7fdc681611c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:52.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.630+0000 7fdc6fada640 1 --2- 172.21.15.60:0/1196077854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc6806f9f0 0x7fdc681f6460 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:52.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.630+0000 7fdc6fada640 1 -- 172.21.15.60:0/1196077854 >> 172.21.15.60:0/1196077854 conn(0x7fdc6809b850 msgr2=0x7fdc68099640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:52.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.630+0000 7fdc6fada640 1 -- 172.21.15.60:0/1196077854 shutdown_connections 2024-05-09T05:01:52.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:52.630+0000 7fdc6fada640 1 -- 172.21.15.60:0/1196077854 wait complete. 2024-05-09T05:01:52.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:52.642 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.pool -v cephfs_data_extra2 dir_x/file_x) 2024-05-09T05:01:52.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:52.661 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -x ceph.dir.layout.pool dir_x) 2024-05-09T05:01:52.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T05:01:52.718 INFO:teuthology.orchestra.run.smithi060.stderr:setfattr: dir_x: No such attribute 2024-05-09T05:01:52.718 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T05:01:52.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T05:01:52.737 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:01:52.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:01:52.811 DEBUG:teuthology.orchestra.run.smithi114:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T05:01:52.859 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T05:01:52.859 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T05:01:52.859 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:01:52.859 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:01:52.876 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:01:52.876 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:01:52.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T05:01:53.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.078+0000 7faa1af40640 1 Processor -- start 2024-05-09T05:01:53.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.078+0000 7faa1af40640 1 -- start start 2024-05-09T05:01:53.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.079+0000 7faa1af40640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faa140b1040 0x7faa140b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.079+0000 7faa1af40640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa140b2d30 0x7faa140b31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.079+0000 7faa1af40640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa1409cd10 con 0x7faa140ae1f0 2024-05-09T05:01:53.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.079+0000 7faa1af40640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7faa140b28a0 con 0x7faa140b1040 2024-05-09T05:01:53.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.079+0000 7faa1af40640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7faa1409d890 con 0x7faa140b2d30 2024-05-09T05:01:53.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.079+0000 7faa0bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faa140b1040 0x7faa140b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.079+0000 7faa18cdc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7faa140ae1f0 0x7faa1409c600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57158/0 (socket says 172.21.15.60:57158) 2024-05-09T05:01:53.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.079+0000 7faa0bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faa140b1040 0x7faa140b14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35674/0 (socket says 172.21.15.60:35674) 2024-05-09T05:01:53.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.079+0000 7faa18cdc640 1 -- 172.21.15.60:0/2064843156 learned_addr learned my addr 172.21.15.60:0/2064843156 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:53.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.079+0000 7faa194dd640 1 --2- 172.21.15.60:0/2064843156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa140b2d30 0x7faa140b31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.079+0000 7faa0bfff640 1 -- 172.21.15.60:0/2064843156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa140b2d30 msgr2=0x7faa140b31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:53.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.079+0000 7faa0bfff640 1 --2- 172.21.15.60:0/2064843156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa140b2d30 0x7faa140b31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.079+0000 7faa0bfff640 1 -- 172.21.15.60:0/2064843156 >> v1:172.21.15.60:6789/0 conn(0x7faa140ae1f0 legacy=0x7faa1409c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:53.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.079+0000 7faa0b7fe640 1 -- 172.21.15.60:0/2064843156 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1827721686 0 0) 0x7faa1409cd10 con 0x7faa140ae1f0 2024-05-09T05:01:53.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.079+0000 7faa0bfff640 1 -- 172.21.15.60:0/2064843156 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa1409d2d0 con 0x7faa140b1040 2024-05-09T05:01:53.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.080+0000 7faa0bfff640 1 --2- 172.21.15.60:0/2064843156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa140b1040 0x7faa140b14c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7faa040097a0 tx=0x7faa0402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d8bb63e48d1c93 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:53.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.080+0000 7faa0b7fe640 1 -- 172.21.15.60:0/2064843156 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa04037070 con 0x7faa140b1040 2024-05-09T05:01:53.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.080+0000 7faa0b7fe640 1 -- 172.21.15.60:0/2064843156 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa04032440 con 0x7faa140b1040 2024-05-09T05:01:53.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.080+0000 7faa0b7fe640 1 -- 172.21.15.60:0/2064843156 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa04032810 con 0x7faa140b1040 2024-05-09T05:01:53.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.080+0000 7faa1af40640 1 -- 172.21.15.60:0/2064843156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa140b1040 msgr2=0x7faa140b14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:53.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.080+0000 7faa1af40640 1 --2- 172.21.15.60:0/2064843156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa140b1040 0x7faa140b14c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7faa040097a0 tx=0x7faa0402a180 comp rx=0 tx=0).stop 2024-05-09T05:01:53.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.081+0000 7faa1af40640 1 -- 172.21.15.60:0/2064843156 shutdown_connections 2024-05-09T05:01:53.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.081+0000 7faa1af40640 1 --2- 172.21.15.60:0/2064843156 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa140b2d30 0x7faa140b31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.081+0000 7faa1af40640 1 --2- 172.21.15.60:0/2064843156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa140b1040 0x7faa140b14c0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.081+0000 7faa1af40640 1 -- 172.21.15.60:0/2064843156 >> 172.21.15.60:0/2064843156 conn(0x7faa14072390 msgr2=0x7faa140727a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:53.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.081+0000 7faa1af40640 1 -- 172.21.15.60:0/2064843156 shutdown_connections 2024-05-09T05:01:53.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.081+0000 7faa1af40640 1 -- 172.21.15.60:0/2064843156 wait complete. 2024-05-09T05:01:53.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.082+0000 7faa1af40640 1 Processor -- start 2024-05-09T05:01:53.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.082+0000 7faa1af40640 1 -- start start 2024-05-09T05:01:53.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.083+0000 7faa1af40640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa140ae1f0 0x7faa1414d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.083+0000 7faa1af40640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa140b1040 0x7faa1415e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.083+0000 7faa1af40640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa140b2d30 0x7faa141f3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.083+0000 7faa1af40640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7faa1409b720 con 0x7faa140ae1f0 2024-05-09T05:01:53.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.083+0000 7faa1af40640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7faa140b2530 con 0x7faa140b2d30 2024-05-09T05:01:53.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.083+0000 7faa1af40640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7faa140b1a00 con 0x7faa140b1040 2024-05-09T05:01:53.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.083+0000 7faa18cdc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa140ae1f0 0x7faa1414d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.083+0000 7faa18cdc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa140ae1f0 0x7faa1414d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35682/0 (socket says 172.21.15.60:35682) 2024-05-09T05:01:53.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.083+0000 7faa18cdc640 1 -- 172.21.15.60:0/3469337226 learned_addr learned my addr 172.21.15.60:0/3469337226 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:53.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.083+0000 7faa0bfff640 1 --2- 172.21.15.60:0/3469337226 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa140b1040 0x7faa1415e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.083+0000 7faa194dd640 1 --2- 172.21.15.60:0/3469337226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa140b2d30 0x7faa141f3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.083+0000 7faa18cdc640 1 -- 172.21.15.60:0/3469337226 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa140b1040 msgr2=0x7faa1415e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:53.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.083+0000 7faa18cdc640 1 --2- 172.21.15.60:0/3469337226 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa140b1040 0x7faa1415e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.083+0000 7faa18cdc640 1 -- 172.21.15.60:0/3469337226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa140b2d30 msgr2=0x7faa141f3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:53.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.083+0000 7faa18cdc640 1 --2- 172.21.15.60:0/3469337226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa140b2d30 0x7faa141f3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.083+0000 7faa18cdc640 1 -- 172.21.15.60:0/3469337226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa14158f20 con 0x7faa140ae1f0 2024-05-09T05:01:53.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.083+0000 7faa194dd640 1 --2- 172.21.15.60:0/3469337226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa140b2d30 0x7faa141f3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:53.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.083+0000 7faa18cdc640 1 --2- 172.21.15.60:0/3469337226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa140ae1f0 0x7faa1414d130 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7faa1000c870 tx=0x7faa1000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:53.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.084+0000 7faa097fa640 1 -- 172.21.15.60:0/3469337226 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa10018070 con 0x7faa140ae1f0 2024-05-09T05:01:53.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.084+0000 7faa097fa640 1 -- 172.21.15.60:0/3469337226 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa1000a5e0 con 0x7faa140ae1f0 2024-05-09T05:01:53.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.084+0000 7faa1af40640 1 -- 172.21.15.60:0/3469337226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa14159200 con 0x7faa140ae1f0 2024-05-09T05:01:53.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.084+0000 7faa097fa640 1 -- 172.21.15.60:0/3469337226 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1000a9d0 con 0x7faa140ae1f0 2024-05-09T05:01:53.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.084+0000 7faa1af40640 1 -- 172.21.15.60:0/3469337226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa14159aa0 con 0x7faa140ae1f0 2024-05-09T05:01:53.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.085+0000 7faa1af40640 1 -- 172.21.15.60:0/3469337226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa141593d0 con 0x7faa140ae1f0 2024-05-09T05:01:53.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.085+0000 7faa097fa640 1 -- 172.21.15.60:0/3469337226 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faa1001b070 con 0x7faa140ae1f0 2024-05-09T05:01:53.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.086+0000 7faa097fa640 1 --2- 172.21.15.60:0/3469337226 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7faa000762d0 0x7faa00078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.086+0000 7faa097fa640 1 -- 172.21.15.60:0/3469337226 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6911+0+0 (secure 0 0 0) 0x7faa10097130 con 0x7faa140ae1f0 2024-05-09T05:01:53.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.086+0000 7faa0bfff640 1 --2- 172.21.15.60:0/3469337226 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7faa000762d0 0x7faa00078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.087+0000 7faa0bfff640 1 --2- 172.21.15.60:0/3469337226 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7faa000762d0 0x7faa00078790 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7faa04034000 tx=0x7faa0402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:53.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.090+0000 7faa097fa640 1 -- 172.21.15.60:0/3469337226 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7faa100605e0 con 0x7faa140ae1f0 2024-05-09T05:01:53.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.292+0000 7faa1af40640 1 -- 172.21.15.60:0/3469337226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7faa14051040 con 0x7faa140ae1f0 2024-05-09T05:01:53.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.292+0000 7faa097fa640 1 -- 172.21.15.60:0/3469337226 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v57) ==== 80+0+0 (secure 0 0 0) 0x7faa1005fee0 con 0x7faa140ae1f0 2024-05-09T05:01:53.293 INFO:teuthology.orchestra.run.smithi060.stderr:listed 0 entries 2024-05-09T05:01:53.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.294+0000 7faa1af40640 1 -- 172.21.15.60:0/3469337226 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7faa000762d0 msgr2=0x7faa00078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:53.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.294+0000 7faa1af40640 1 --2- 172.21.15.60:0/3469337226 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7faa000762d0 0x7faa00078790 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7faa04034000 tx=0x7faa0402e000 comp rx=0 tx=0).stop 2024-05-09T05:01:53.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.294+0000 7faa1af40640 1 -- 172.21.15.60:0/3469337226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa140ae1f0 msgr2=0x7faa1414d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:53.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.294+0000 7faa1af40640 1 --2- 172.21.15.60:0/3469337226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa140ae1f0 0x7faa1414d130 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7faa1000c870 tx=0x7faa1000cd30 comp rx=0 tx=0).stop 2024-05-09T05:01:53.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.294+0000 7faa1af40640 1 -- 172.21.15.60:0/3469337226 shutdown_connections 2024-05-09T05:01:53.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.294+0000 7faa1af40640 1 --2- 172.21.15.60:0/3469337226 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7faa000762d0 0x7faa00078790 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.294+0000 7faa1af40640 1 --2- 172.21.15.60:0/3469337226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa140b2d30 0x7faa141f3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.294+0000 7faa1af40640 1 --2- 172.21.15.60:0/3469337226 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7faa140b1040 0x7faa1415e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.294+0000 7faa1af40640 1 --2- 172.21.15.60:0/3469337226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa140ae1f0 0x7faa1414d130 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.294+0000 7faa1af40640 1 -- 172.21.15.60:0/3469337226 >> 172.21.15.60:0/3469337226 conn(0x7faa14072390 msgr2=0x7faa140ac240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:53.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.294+0000 7faa1af40640 1 -- 172.21.15.60:0/3469337226 shutdown_connections 2024-05-09T05:01:53.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.294+0000 7faa1af40640 1 -- 172.21.15.60:0/3469337226 wait complete. 2024-05-09T05:01:53.306 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-09T05:01:53.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:53.306 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-09T05:01:53.361 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2024-05-09T05:01:53.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:01:53.362 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T05:01:53.389 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T05:01:53.390 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T05:01:53.390 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:01:53.407 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056322,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497656k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248824k,nr_inodes=812206,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497656k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T05:01:53.407 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T05:01:53.408 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:01:53.408 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:01:53.447 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057102,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498908k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19678 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498908k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T05:01:53.447 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T05:01:53.447 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T05:01:53.447 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:01:53.465 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056322,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497656k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248824k,nr_inodes=812206,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497656k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T05:01:53.465 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:53.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.590+0000 7efc0da93640 1 Processor -- start 2024-05-09T05:01:53.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.591+0000 7efc0da93640 1 -- start start 2024-05-09T05:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.591+0000 7efc0da93640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7efc081606f0 0x7efc080d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.591+0000 7efc0da93640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc0806ff60 con 0x7efc0815f300 2024-05-09T05:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.591+0000 7efc0da93640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc08070240 con 0x7efc0806f9f0 2024-05-09T05:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.591+0000 7efc0da93640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7efc08070520 con 0x7efc081606f0 2024-05-09T05:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.591+0000 7efc077fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efc0815f300 0x7efc081639f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57164/0 (socket says 172.21.15.60:57164) 2024-05-09T05:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.591+0000 7efc077fe640 1 -- 172.21.15.60:0/3821530576 learned_addr learned my addr 172.21.15.60:0/3821530576 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc06ffd640 1 --2- 172.21.15.60:0/3821530576 >> v2:172.21.15.114:3300/0 conn(0x7efc081606f0 0x7efc080d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc067fc640 1 -- 172.21.15.60:0/3821530576 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 574774554 0 0) 0x7efc0806ff60 con 0x7efc0815f300 2024-05-09T05:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc067fc640 1 -- 172.21.15.60:0/3821530576 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efbf0003520 con 0x7efc0815f300 2024-05-09T05:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc067fc640 1 -- 172.21.15.60:0/3821530576 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3914776966 0 0) 0x7efbf0003520 con 0x7efc0815f300 2024-05-09T05:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc067fc640 1 -- 172.21.15.60:0/3821530576 >> v2:172.21.15.114:3300/0 conn(0x7efc081606f0 msgr2=0x7efc080d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc067fc640 1 --2- 172.21.15.60:0/3821530576 >> v2:172.21.15.114:3300/0 conn(0x7efc081606f0 0x7efc080d33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc067fc640 1 -- 172.21.15.60:0/3821530576 >> v1:172.21.15.114:6789/0 conn(0x7efc0806f9f0 legacy=0x7efc0806ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:53.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc067fc640 1 -- 172.21.15.60:0/3821530576 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc0806e740 con 0x7efc0815f300 2024-05-09T05:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc067fc640 1 -- 172.21.15.60:0/3821530576 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7efbf80016f0 con 0x7efc0815f300 2024-05-09T05:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc067fc640 1 -- 172.21.15.60:0/3821530576 >> v1:172.21.15.60:6789/0 conn(0x7efc0815f300 legacy=0x7efc081639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc067fc640 1 --2- 172.21.15.60:0/3821530576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efbf0003a50 0x7efbf0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc067fc640 1 --2- 172.21.15.60:0/3821530576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efbf0004800 0x7efbf0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc067fc640 1 --2- 172.21.15.60:0/3821530576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efbf00051c0 0x7efbf000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc067fc640 1 -- 172.21.15.60:0/3821530576 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efc08070520 con 0x7efbf0004800 2024-05-09T05:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc067fc640 1 -- 172.21.15.60:0/3821530576 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efbf000c160 con 0x7efbf0003a50 2024-05-09T05:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc067fc640 1 -- 172.21.15.60:0/3821530576 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7efbf000c360 con 0x7efbf00051c0 2024-05-09T05:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc06ffd640 1 --2- 172.21.15.60:0/3821530576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efbf0004800 0x7efbf0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc077fe640 1 --2- 172.21.15.60:0/3821530576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efbf0003a50 0x7efbf0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc07fff640 1 --2- 172.21.15.60:0/3821530576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efbf00051c0 0x7efbf000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc06ffd640 1 -- 172.21.15.60:0/3821530576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efbf00051c0 msgr2=0x7efbf000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc06ffd640 1 --2- 172.21.15.60:0/3821530576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efbf00051c0 0x7efbf000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc06ffd640 1 -- 172.21.15.60:0/3821530576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efbf0003a50 msgr2=0x7efbf0003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.592+0000 7efc06ffd640 1 --2- 172.21.15.60:0/3821530576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efbf0003a50 0x7efbf0003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.593+0000 7efc07fff640 1 -- 172.21.15.60:0/3821530576 reap_dead start 2024-05-09T05:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.593+0000 7efc06ffd640 1 -- 172.21.15.60:0/3821530576 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbf0003520 con 0x7efbf0004800 2024-05-09T05:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.593+0000 7efc06ffd640 1 --2- 172.21.15.60:0/3821530576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efbf0004800 0x7efbf0004c80 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7efbfc009a90 tx=0x7efbf0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff87bd0853136de server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.593+0000 7efc067fc640 1 -- 172.21.15.60:0/3821530576 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc036070 con 0x7efbf0004800 2024-05-09T05:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.593+0000 7efc067fc640 1 -- 172.21.15.60:0/3821530576 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbfc031400 con 0x7efbf0004800 2024-05-09T05:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.593+0000 7efc067fc640 1 -- 172.21.15.60:0/3821530576 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc009690 con 0x7efbf0004800 2024-05-09T05:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.594+0000 7efc0da93640 1 -- 172.21.15.60:0/3821530576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efbf0004800 msgr2=0x7efbf0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.594+0000 7efc0da93640 1 --2- 172.21.15.60:0/3821530576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efbf0004800 0x7efbf0004c80 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7efbfc009a90 tx=0x7efbf0005830 comp rx=0 tx=0).stop 2024-05-09T05:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.594+0000 7efc0da93640 1 -- 172.21.15.60:0/3821530576 shutdown_connections 2024-05-09T05:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.594+0000 7efc0da93640 1 --2- 172.21.15.60:0/3821530576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efbf0004800 0x7efbf0004c80 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.594+0000 7efc0da93640 1 -- 172.21.15.60:0/3821530576 >> 172.21.15.60:0/3821530576 conn(0x7efc0809b850 msgr2=0x7efc0809bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:53.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.594+0000 7efc0da93640 1 -- 172.21.15.60:0/3821530576 shutdown_connections 2024-05-09T05:01:53.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.594+0000 7efc0da93640 1 -- 172.21.15.60:0/3821530576 wait complete. 2024-05-09T05:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.595+0000 7efc0da93640 1 Processor -- start 2024-05-09T05:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.595+0000 7efc0da93640 1 -- start start 2024-05-09T05:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.596+0000 7efc0da93640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efbf0004800 0x7efc081f5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.596+0000 7efc0da93640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc080af530 0x7efc0815e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.596+0000 7efc0da93640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc080b0a60 0x7efc080b1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.596+0000 7efc0da93640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7efc08163b70 con 0x7efc080b0a60 2024-05-09T05:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.596+0000 7efc0da93640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efc081639f0 con 0x7efbf0004800 2024-05-09T05:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.596+0000 7efc0da93640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7efc08163cf0 con 0x7efc080af530 2024-05-09T05:01:53.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.596+0000 7efc07fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc080b0a60 0x7efc080b1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.596+0000 7efc07fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc080b0a60 0x7efc080b1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35692/0 (socket says 172.21.15.60:35692) 2024-05-09T05:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.596+0000 7efc07fff640 1 -- 172.21.15.60:0/1814607305 learned_addr learned my addr 172.21.15.60:0/1814607305 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.596+0000 7efc06ffd640 1 --2- 172.21.15.60:0/1814607305 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc080af530 0x7efc0815e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.596+0000 7efc077fe640 1 --2- 172.21.15.60:0/1814607305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efbf0004800 0x7efc081f5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.596+0000 7efc07fff640 1 -- 172.21.15.60:0/1814607305 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc080af530 msgr2=0x7efc0815e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.596+0000 7efc07fff640 1 --2- 172.21.15.60:0/1814607305 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc080af530 0x7efc0815e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.596+0000 7efc07fff640 1 -- 172.21.15.60:0/1814607305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efbf0004800 msgr2=0x7efc081f5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.596+0000 7efc07fff640 1 --2- 172.21.15.60:0/1814607305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efbf0004800 0x7efc081f5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.596+0000 7efc07fff640 1 -- 172.21.15.60:0/1814607305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc080af9c0 con 0x7efc080b0a60 2024-05-09T05:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.596+0000 7efc06ffd640 1 --2- 172.21.15.60:0/1814607305 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc080af530 0x7efc0815e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.596+0000 7efc077fe640 1 --2- 172.21.15.60:0/1814607305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efbf0004800 0x7efc081f5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.597+0000 7efc07fff640 1 --2- 172.21.15.60:0/1814607305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc080b0a60 0x7efc080b1e70 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7efc08070830 tx=0x7efc0000fd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.597+0000 7efbeffff640 1 -- 172.21.15.60:0/1814607305 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc00019070 con 0x7efc080b0a60 2024-05-09T05:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.597+0000 7efbeffff640 1 -- 172.21.15.60:0/1814607305 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc00006d60 con 0x7efc080b0a60 2024-05-09T05:01:53.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.597+0000 7efc0da93640 1 -- 172.21.15.60:0/1814607305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc080afca0 con 0x7efc080b0a60 2024-05-09T05:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.597+0000 7efbeffff640 1 -- 172.21.15.60:0/1814607305 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc00003bb0 con 0x7efc080b0a60 2024-05-09T05:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.597+0000 7efc0da93640 1 -- 172.21.15.60:0/1814607305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc080b0260 con 0x7efc080b0a60 2024-05-09T05:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.598+0000 7efc0da93640 1 -- 172.21.15.60:0/1814607305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc08051040 con 0x7efc080b0a60 2024-05-09T05:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.598+0000 7efbeffff640 1 -- 172.21.15.60:0/1814607305 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efc0001c070 con 0x7efc080b0a60 2024-05-09T05:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.599+0000 7efbeffff640 1 --2- 172.21.15.60:0/1814607305 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7efbf40762d0 0x7efbf4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.599+0000 7efbeffff640 1 -- 172.21.15.60:0/1814607305 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6911+0+0 (secure 0 0 0) 0x7efc00096e40 con 0x7efc080b0a60 2024-05-09T05:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.599+0000 7efc077fe640 1 --2- 172.21.15.60:0/1814607305 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7efbf40762d0 0x7efbf4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.600+0000 7efc077fe640 1 --2- 172.21.15.60:0/1814607305 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7efbf40762d0 0x7efbf4078790 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efbf8001010 tx=0x7efbf8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:53.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.603+0000 7efbeffff640 1 -- 172.21.15.60:0/1814607305 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7efc00060250 con 0x7efc080b0a60 2024-05-09T05:01:53.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.832+0000 7efc0da93640 1 -- 172.21.15.60:0/1814607305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efc0809f8e0 con 0x7efc080b0a60 2024-05-09T05:01:53.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.832+0000 7efbeffff640 1 -- 172.21.15.60:0/1814607305 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+5193 (secure 0 0 0) 0x7efc0005fb50 con 0x7efc080b0a60 2024-05-09T05:01:53.833 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:53.833 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":29,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3604145570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3604145570},{"type":"v1","addr":"172.21.15.60:6834","nonce":3604145570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3949194455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3949194455},{"type":"v1","addr":"172.21.15.114:6836","nonce":3949194455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3288630382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3288630382},{"type":"v1","addr":"172.21.15.114:6837","nonce":3288630382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:39.237498+0000","modified":"2024-05-09T05:01:51.305520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5415":{"gid":5415,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2425263408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2425263408},{"type":"v1","addr":"172.21.15.60:6835","nonce":2425263408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9,10,11],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5415,"qdb_cluster":[5415]},"id":3}]} 2024-05-09T05:01:53.833 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 29 2024-05-09T05:01:53.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.834+0000 7efc0da93640 1 -- 172.21.15.60:0/1814607305 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7efbf40762d0 msgr2=0x7efbf4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:53.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.834+0000 7efc0da93640 1 --2- 172.21.15.60:0/1814607305 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7efbf40762d0 0x7efbf4078790 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efbf8001010 tx=0x7efbf8009040 comp rx=0 tx=0).stop 2024-05-09T05:01:53.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.834+0000 7efc0da93640 1 -- 172.21.15.60:0/1814607305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc080b0a60 msgr2=0x7efc080b1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:53.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.834+0000 7efc0da93640 1 --2- 172.21.15.60:0/1814607305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc080b0a60 0x7efc080b1e70 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7efc08070830 tx=0x7efc0000fd20 comp rx=0 tx=0).stop 2024-05-09T05:01:53.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.834+0000 7efc0da93640 1 -- 172.21.15.60:0/1814607305 shutdown_connections 2024-05-09T05:01:53.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.834+0000 7efc0da93640 1 --2- 172.21.15.60:0/1814607305 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7efbf40762d0 0x7efbf4078790 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.834+0000 7efc0da93640 1 --2- 172.21.15.60:0/1814607305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc080b0a60 0x7efc080b1e70 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.834+0000 7efc0da93640 1 --2- 172.21.15.60:0/1814607305 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7efc080af530 0x7efc0815e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.834+0000 7efc0da93640 1 --2- 172.21.15.60:0/1814607305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efbf0004800 0x7efc081f5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.834+0000 7efc0da93640 1 -- 172.21.15.60:0/1814607305 >> 172.21.15.60:0/1814607305 conn(0x7efc0809b850 msgr2=0x7efc080998a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:53.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.834+0000 7efc0da93640 1 -- 172.21.15.60:0/1814607305 shutdown_connections 2024-05-09T05:01:53.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.834+0000 7efc0da93640 1 -- 172.21.15.60:0/1814607305 wait complete. 2024-05-09T05:01:53.846 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:53.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.949+0000 7fbace009640 1 Processor -- start 2024-05-09T05:01:53.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.949+0000 7fbace009640 1 -- start start 2024-05-09T05:01:53.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.949+0000 7fbace009640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbac81606f0 0x7fbac80d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.949+0000 7fbace009640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac806f9f0 0x7fbac80d3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbace009640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbac806ff60 con 0x7fbac815f300 2024-05-09T05:01:53.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbace009640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fbac8070240 con 0x7fbac81606f0 2024-05-09T05:01:53.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbace009640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbac8070520 con 0x7fbac806f9f0 2024-05-09T05:01:53.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac77fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbac815f300 0x7fbac81639f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57180/0 (socket says 172.21.15.60:57180) 2024-05-09T05:01:53.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac77fe640 1 -- 172.21.15.60:0/1801935633 learned_addr learned my addr 172.21.15.60:0/1801935633 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:53.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac7fff640 1 --2- 172.21.15.60:0/1801935633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac806f9f0 0x7fbac80d3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac67fc640 1 -- 172.21.15.60:0/1801935633 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4214565897 0 0) 0x7fbac806ff60 con 0x7fbac815f300 2024-05-09T05:01:53.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac67fc640 1 -- 172.21.15.60:0/1801935633 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbab0003520 con 0x7fbac815f300 2024-05-09T05:01:53.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac6ffd640 1 --2- 172.21.15.60:0/1801935633 >> v2:172.21.15.114:3300/0 conn(0x7fbac81606f0 0x7fbac80d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac67fc640 1 -- 172.21.15.60:0/1801935633 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3082931413 0 0) 0x7fbab0003520 con 0x7fbac815f300 2024-05-09T05:01:53.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac67fc640 1 -- 172.21.15.60:0/1801935633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac806f9f0 msgr2=0x7fbac80d3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:53.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac67fc640 1 --2- 172.21.15.60:0/1801935633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac806f9f0 0x7fbac80d3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac67fc640 1 -- 172.21.15.60:0/1801935633 >> v2:172.21.15.114:3300/0 conn(0x7fbac81606f0 msgr2=0x7fbac80d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:53.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac67fc640 1 --2- 172.21.15.60:0/1801935633 >> v2:172.21.15.114:3300/0 conn(0x7fbac81606f0 0x7fbac80d33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac67fc640 1 -- 172.21.15.60:0/1801935633 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac806e740 con 0x7fbac815f300 2024-05-09T05:01:53.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac67fc640 1 -- 172.21.15.60:0/1801935633 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7fbab80016f0 con 0x7fbac815f300 2024-05-09T05:01:53.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac67fc640 1 -- 172.21.15.60:0/1801935633 >> v1:172.21.15.60:6789/0 conn(0x7fbac815f300 legacy=0x7fbac81639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:53.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac67fc640 1 --2- 172.21.15.60:0/1801935633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbab0003a00 0x7fbab0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac67fc640 1 --2- 172.21.15.60:0/1801935633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac806f9f0 0x7fbab0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac67fc640 1 --2- 172.21.15.60:0/1801935633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbab0004d40 0x7fbab00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac67fc640 1 -- 172.21.15.60:0/1801935633 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbac8070240 con 0x7fbab0004d40 2024-05-09T05:01:53.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac67fc640 1 -- 172.21.15.60:0/1801935633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbac8070520 con 0x7fbab0003a00 2024-05-09T05:01:53.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac67fc640 1 -- 172.21.15.60:0/1801935633 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbab000b9f0 con 0x7fbac806f9f0 2024-05-09T05:01:53.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac7fff640 1 --2- 172.21.15.60:0/1801935633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbab0004d40 0x7fbab00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac77fe640 1 --2- 172.21.15.60:0/1801935633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbab0003a00 0x7fbab0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.950+0000 7fbac6ffd640 1 --2- 172.21.15.60:0/1801935633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac806f9f0 0x7fbab0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.951+0000 7fbac7fff640 1 -- 172.21.15.60:0/1801935633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac806f9f0 msgr2=0x7fbab0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:53.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.951+0000 7fbac7fff640 1 --2- 172.21.15.60:0/1801935633 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac806f9f0 0x7fbab0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.951+0000 7fbac7fff640 1 -- 172.21.15.60:0/1801935633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbab0003a00 msgr2=0x7fbab0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:53.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.951+0000 7fbac7fff640 1 --2- 172.21.15.60:0/1801935633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbab0003a00 0x7fbab0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.951+0000 7fbac7fff640 1 -- 172.21.15.60:0/1801935633 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbab0003520 con 0x7fbab0004d40 2024-05-09T05:01:53.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.951+0000 7fbac7fff640 1 --2- 172.21.15.60:0/1801935633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbab0004d40 0x7fbab00051c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbac0009fc0 tx=0x7fbab0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1712cc17c9dfa042 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:53.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.952+0000 7fbac67fc640 1 -- 172.21.15.60:0/1801935633 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0009c10 con 0x7fbab0004d40 2024-05-09T05:01:53.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.952+0000 7fbac67fc640 1 -- 172.21.15.60:0/1801935633 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbac0036070 con 0x7fbab0004d40 2024-05-09T05:01:53.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.952+0000 7fbac67fc640 1 -- 172.21.15.60:0/1801935633 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0031880 con 0x7fbab0004d40 2024-05-09T05:01:53.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.952+0000 7fbace009640 1 -- 172.21.15.60:0/1801935633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbab0004d40 msgr2=0x7fbab00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:53.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.952+0000 7fbace009640 1 --2- 172.21.15.60:0/1801935633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbab0004d40 0x7fbab00051c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbac0009fc0 tx=0x7fbab0005830 comp rx=0 tx=0).stop 2024-05-09T05:01:53.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.952+0000 7fbac7fff640 1 -- 172.21.15.60:0/1801935633 reap_dead start 2024-05-09T05:01:53.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.952+0000 7fbace009640 1 -- 172.21.15.60:0/1801935633 shutdown_connections 2024-05-09T05:01:53.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.952+0000 7fbace009640 1 -- 172.21.15.60:0/1801935633 >> 172.21.15.60:0/1801935633 conn(0x7fbac809b850 msgr2=0x7fbac809bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:53.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.952+0000 7fbace009640 1 -- 172.21.15.60:0/1801935633 shutdown_connections 2024-05-09T05:01:53.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.952+0000 7fbace009640 1 -- 172.21.15.60:0/1801935633 wait complete. 2024-05-09T05:01:53.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.953+0000 7fbace009640 1 Processor -- start 2024-05-09T05:01:53.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.954+0000 7fbace009640 1 -- start start 2024-05-09T05:01:53.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.954+0000 7fbace009640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac8159860 0x7fbac809f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.954+0000 7fbace009640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac815cf90 0x7fbac815d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.954+0000 7fbace009640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbac81f5370 0x7fbac81f57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.954+0000 7fbace009640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbac80d3ab0 con 0x7fbac81f5370 2024-05-09T05:01:53.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.954+0000 7fbace009640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbac80d3930 con 0x7fbac815cf90 2024-05-09T05:01:53.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.954+0000 7fbace009640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbac80d3c30 con 0x7fbac8159860 2024-05-09T05:01:53.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.955+0000 7fbac7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbac81f5370 0x7fbac81f57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.955+0000 7fbac7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbac81f5370 0x7fbac81f57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35720/0 (socket says 172.21.15.60:35720) 2024-05-09T05:01:53.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.955+0000 7fbac7fff640 1 -- 172.21.15.60:0/458694606 learned_addr learned my addr 172.21.15.60:0/458694606 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:53.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.955+0000 7fbac77fe640 1 --2- 172.21.15.60:0/458694606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac8159860 0x7fbac809f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.955+0000 7fbac6ffd640 1 --2- 172.21.15.60:0/458694606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac815cf90 0x7fbac815d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.955+0000 7fbac7fff640 1 -- 172.21.15.60:0/458694606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac8159860 msgr2=0x7fbac809f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:53.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.955+0000 7fbac7fff640 1 --2- 172.21.15.60:0/458694606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac8159860 0x7fbac809f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.955+0000 7fbac7fff640 1 -- 172.21.15.60:0/458694606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac815cf90 msgr2=0x7fbac815d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:53.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.955+0000 7fbac7fff640 1 --2- 172.21.15.60:0/458694606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac815cf90 0x7fbac815d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:53.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.955+0000 7fbac7fff640 1 -- 172.21.15.60:0/458694606 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac815a2b0 con 0x7fbac81f5370 2024-05-09T05:01:53.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.955+0000 7fbac7fff640 1 --2- 172.21.15.60:0/458694606 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbac81f5370 0x7fbac81f57f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fbac002aee0 tx=0x7fbac0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:53.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.955+0000 7fbaaffff640 1 -- 172.21.15.60:0/458694606 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0009c10 con 0x7fbac81f5370 2024-05-09T05:01:53.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.955+0000 7fbaaffff640 1 -- 172.21.15.60:0/458694606 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbac0040030 con 0x7fbac81f5370 2024-05-09T05:01:53.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.955+0000 7fbace009640 1 -- 172.21.15.60:0/458694606 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbac809e170 con 0x7fbac81f5370 2024-05-09T05:01:53.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.955+0000 7fbaaffff640 1 -- 172.21.15.60:0/458694606 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0003bd0 con 0x7fbac81f5370 2024-05-09T05:01:53.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.955+0000 7fbace009640 1 -- 172.21.15.60:0/458694606 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbac809e730 con 0x7fbac81f5370 2024-05-09T05:01:53.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.956+0000 7fbace009640 1 -- 172.21.15.60:0/458694606 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbac8051040 con 0x7fbac81f5370 2024-05-09T05:01:53.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.957+0000 7fbaaffff640 1 -- 172.21.15.60:0/458694606 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbac0005d10 con 0x7fbac81f5370 2024-05-09T05:01:53.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.957+0000 7fbaaffff640 1 --2- 172.21.15.60:0/458694606 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbab40762d0 0x7fbab4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:53.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.957+0000 7fbaaffff640 1 -- 172.21.15.60:0/458694606 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6911+0+0 (secure 0 0 0) 0x7fbac00b8e00 con 0x7fbac81f5370 2024-05-09T05:01:53.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.958+0000 7fbac77fe640 1 --2- 172.21.15.60:0/458694606 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbab40762d0 0x7fbab4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:53.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.958+0000 7fbac77fe640 1 --2- 172.21.15.60:0/458694606 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbab40762d0 0x7fbab4078790 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbab8001eb0 tx=0x7fbab8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:53.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:53.962+0000 7fbaaffff640 1 -- 172.21.15.60:0/458694606 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbac004a020 con 0x7fbac81f5370 2024-05-09T05:01:54.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.189+0000 7fbace009640 1 -- 172.21.15.60:0/458694606 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbac8073820 con 0x7fbac81f5370 2024-05-09T05:01:54.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.189+0000 7fbaaffff640 1 -- 172.21.15.60:0/458694606 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+5193 (secure 0 0 0) 0x7fbac008a020 con 0x7fbac81f5370 2024-05-09T05:01:54.190 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:54.190 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":29,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3604145570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3604145570},{"type":"v1","addr":"172.21.15.60:6834","nonce":3604145570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3949194455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3949194455},{"type":"v1","addr":"172.21.15.114:6836","nonce":3949194455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3288630382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3288630382},{"type":"v1","addr":"172.21.15.114:6837","nonce":3288630382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:39.237498+0000","modified":"2024-05-09T05:01:51.305520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5415":{"gid":5415,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2425263408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2425263408},{"type":"v1","addr":"172.21.15.60:6835","nonce":2425263408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9,10,11],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5415,"qdb_cluster":[5415]},"id":3}]} 2024-05-09T05:01:54.190 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 29 2024-05-09T05:01:54.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.191+0000 7fbace009640 1 -- 172.21.15.60:0/458694606 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbab40762d0 msgr2=0x7fbab4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:54.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.191+0000 7fbace009640 1 --2- 172.21.15.60:0/458694606 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbab40762d0 0x7fbab4078790 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbab8001eb0 tx=0x7fbab8009040 comp rx=0 tx=0).stop 2024-05-09T05:01:54.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.191+0000 7fbace009640 1 -- 172.21.15.60:0/458694606 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbac81f5370 msgr2=0x7fbac81f57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:54.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.191+0000 7fbace009640 1 --2- 172.21.15.60:0/458694606 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbac81f5370 0x7fbac81f57f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fbac002aee0 tx=0x7fbac0009750 comp rx=0 tx=0).stop 2024-05-09T05:01:54.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.191+0000 7fbace009640 1 -- 172.21.15.60:0/458694606 shutdown_connections 2024-05-09T05:01:54.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.191+0000 7fbace009640 1 --2- 172.21.15.60:0/458694606 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbab40762d0 0x7fbab4078790 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.191+0000 7fbace009640 1 --2- 172.21.15.60:0/458694606 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbac81f5370 0x7fbac81f57f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.191+0000 7fbace009640 1 --2- 172.21.15.60:0/458694606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbac815cf90 0x7fbac815d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.191+0000 7fbace009640 1 --2- 172.21.15.60:0/458694606 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbac8159860 0x7fbac809f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.191+0000 7fbace009640 1 -- 172.21.15.60:0/458694606 >> 172.21.15.60:0/458694606 conn(0x7fbac809b850 msgr2=0x7fbac809a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:54.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.192+0000 7fbace009640 1 -- 172.21.15.60:0/458694606 shutdown_connections 2024-05-09T05:01:54.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.192+0000 7fbace009640 1 -- 172.21.15.60:0/458694606 wait complete. 2024-05-09T05:01:54.203 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:01:54.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.306+0000 7fbce9c93640 1 Processor -- start 2024-05-09T05:01:54.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.306+0000 7fbce9c93640 1 -- start start 2024-05-09T05:01:54.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.307+0000 7fbce9c93640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbce40b1700 0x7fbce4098760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:54.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.307+0000 7fbce9c93640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbce40b34b0 0x7fbce4098ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:54.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.307+0000 7fbce9c93640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbce40aed80 con 0x7fbce4163480 2024-05-09T05:01:54.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.307+0000 7fbce9c93640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fbce40af060 con 0x7fbce40b1700 2024-05-09T05:01:54.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.307+0000 7fbce9c93640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbce40b29d0 con 0x7fbce40b34b0 2024-05-09T05:01:54.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.307+0000 7fbce37fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbce4163480 0x7fbce4163860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57184/0 (socket says 172.21.15.60:57184) 2024-05-09T05:01:54.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.307+0000 7fbce37fe640 1 -- 172.21.15.60:0/2128164057 learned_addr learned my addr 172.21.15.60:0/2128164057 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:54.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.307+0000 7fbce2ffd640 1 --2- 172.21.15.60:0/2128164057 >> v2:172.21.15.114:3300/0 conn(0x7fbce40b1700 0x7fbce4098760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:54.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.307+0000 7fbce3fff640 1 --2- 172.21.15.60:0/2128164057 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbce40b34b0 0x7fbce4098ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:54.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.307+0000 7fbce27fc640 1 -- 172.21.15.60:0/2128164057 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3755674397 0 0) 0x7fbce40aed80 con 0x7fbce4163480 2024-05-09T05:01:54.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.307+0000 7fbce27fc640 1 -- 172.21.15.60:0/2128164057 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbccc003520 con 0x7fbce4163480 2024-05-09T05:01:54.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.307+0000 7fbce3fff640 1 -- 172.21.15.60:0/2128164057 >> v2:172.21.15.114:3300/0 conn(0x7fbce40b1700 msgr2=0x7fbce4098760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:54.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.307+0000 7fbce3fff640 1 --2- 172.21.15.60:0/2128164057 >> v2:172.21.15.114:3300/0 conn(0x7fbce40b1700 0x7fbce4098760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.307+0000 7fbce3fff640 1 -- 172.21.15.60:0/2128164057 >> v1:172.21.15.60:6789/0 conn(0x7fbce4163480 legacy=0x7fbce4163860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:54.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.307+0000 7fbce3fff640 1 -- 172.21.15.60:0/2128164057 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce40b2cb0 con 0x7fbce40b34b0 2024-05-09T05:01:54.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.308+0000 7fbce3fff640 1 --2- 172.21.15.60:0/2128164057 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbce40b34b0 0x7fbce4098ca0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc009d90 tx=0x7fbcdc02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd8bde1a5d1892c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:54.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.308+0000 7fbce27fc640 1 -- 172.21.15.60:0/2128164057 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc007740 con 0x7fbce40b34b0 2024-05-09T05:01:54.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.308+0000 7fbce27fc640 1 -- 172.21.15.60:0/2128164057 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcdc0078e0 con 0x7fbce40b34b0 2024-05-09T05:01:54.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.308+0000 7fbce27fc640 1 -- 172.21.15.60:0/2128164057 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc007cb0 con 0x7fbce40b34b0 2024-05-09T05:01:54.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.309+0000 7fbce9c93640 1 -- 172.21.15.60:0/2128164057 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbce40b34b0 msgr2=0x7fbce4098ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:54.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.309+0000 7fbce9c93640 1 --2- 172.21.15.60:0/2128164057 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbce40b34b0 0x7fbce4098ca0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc009d90 tx=0x7fbcdc02a660 comp rx=0 tx=0).stop 2024-05-09T05:01:54.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.309+0000 7fbce9c93640 1 -- 172.21.15.60:0/2128164057 shutdown_connections 2024-05-09T05:01:54.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.309+0000 7fbce9c93640 1 --2- 172.21.15.60:0/2128164057 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbce40b34b0 0x7fbce4098ca0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.309+0000 7fbce9c93640 1 --2- 172.21.15.60:0/2128164057 >> v2:172.21.15.114:3300/0 conn(0x7fbce40b1700 0x7fbce4098760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.309+0000 7fbce9c93640 1 -- 172.21.15.60:0/2128164057 >> 172.21.15.60:0/2128164057 conn(0x7fbce40b42e0 msgr2=0x7fbce40b46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:54.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.309+0000 7fbce9c93640 1 -- 172.21.15.60:0/2128164057 shutdown_connections 2024-05-09T05:01:54.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.309+0000 7fbce9c93640 1 -- 172.21.15.60:0/2128164057 wait complete. 2024-05-09T05:01:54.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.310+0000 7fbce9c93640 1 Processor -- start 2024-05-09T05:01:54.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.310+0000 7fbce9c93640 1 -- start start 2024-05-09T05:01:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.311+0000 7fbce9c93640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbce40b1700 0x7fbce406ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.311+0000 7fbce9c93640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbce40b34b0 0x7fbce41f4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.311+0000 7fbce9c93640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbce4163480 0x7fbce41f4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.311+0000 7fbce9c93640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbce40ae910 con 0x7fbce4163480 2024-05-09T05:01:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.311+0000 7fbce9c93640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbce4163f70 con 0x7fbce40b34b0 2024-05-09T05:01:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.311+0000 7fbce9c93640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbce40aea90 con 0x7fbce40b1700 2024-05-09T05:01:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.311+0000 7fbce3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbce4163480 0x7fbce41f4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.311+0000 7fbce3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbce4163480 0x7fbce41f4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35726/0 (socket says 172.21.15.60:35726) 2024-05-09T05:01:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.311+0000 7fbce3fff640 1 -- 172.21.15.60:0/992488383 learned_addr learned my addr 172.21.15.60:0/992488383 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.311+0000 7fbce37fe640 1 --2- 172.21.15.60:0/992488383 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbce40b1700 0x7fbce406ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.311+0000 7fbce2ffd640 1 --2- 172.21.15.60:0/992488383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbce40b34b0 0x7fbce41f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.312+0000 7fbce3fff640 1 -- 172.21.15.60:0/992488383 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbce40b1700 msgr2=0x7fbce406ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.312+0000 7fbce3fff640 1 --2- 172.21.15.60:0/992488383 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbce40b1700 0x7fbce406ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.312+0000 7fbce3fff640 1 -- 172.21.15.60:0/992488383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbce40b34b0 msgr2=0x7fbce41f4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.312+0000 7fbce3fff640 1 --2- 172.21.15.60:0/992488383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbce40b34b0 0x7fbce41f4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.312+0000 7fbce3fff640 1 -- 172.21.15.60:0/992488383 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce41f4cd0 con 0x7fbce4163480 2024-05-09T05:01:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.312+0000 7fbce3fff640 1 --2- 172.21.15.60:0/992488383 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbce4163480 0x7fbce41f4790 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc009eb0 tx=0x7fbcdc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.312+0000 7fbccbfff640 1 -- 172.21.15.60:0/992488383 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc040070 con 0x7fbce4163480 2024-05-09T05:01:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.312+0000 7fbccbfff640 1 -- 172.21.15.60:0/992488383 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcdc031bc0 con 0x7fbce4163480 2024-05-09T05:01:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.312+0000 7fbce9c93640 1 -- 172.21.15.60:0/992488383 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbce4162360 con 0x7fbce4163480 2024-05-09T05:01:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.312+0000 7fbccbfff640 1 -- 172.21.15.60:0/992488383 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc009450 con 0x7fbce4163480 2024-05-09T05:01:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.312+0000 7fbce9c93640 1 -- 172.21.15.60:0/992488383 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbce4162c00 con 0x7fbce4163480 2024-05-09T05:01:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.313+0000 7fbce9c93640 1 -- 172.21.15.60:0/992488383 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbce4162750 con 0x7fbce4163480 2024-05-09T05:01:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.314+0000 7fbccbfff640 1 -- 172.21.15.60:0/992488383 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbcdc039040 con 0x7fbce4163480 2024-05-09T05:01:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.314+0000 7fbccbfff640 1 --2- 172.21.15.60:0/992488383 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbcd00762d0 0x7fbcd0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.314+0000 7fbccbfff640 1 -- 172.21.15.60:0/992488383 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6911+0+0 (secure 0 0 0) 0x7fbcdc0b9180 con 0x7fbce4163480 2024-05-09T05:01:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.315+0000 7fbce37fe640 1 --2- 172.21.15.60:0/992488383 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbcd00762d0 0x7fbcd0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.315+0000 7fbce37fe640 1 --2- 172.21.15.60:0/992488383 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbcd00762d0 0x7fbcd0078790 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4007f50 tx=0x7fbcd4007ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:54.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.318+0000 7fbccbfff640 1 -- 172.21.15.60:0/992488383 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbcdc081fe0 con 0x7fbce4163480 2024-05-09T05:01:54.511 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.511+0000 7fbce9c93640 1 -- 172.21.15.60:0/992488383 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbce406d410 con 0x7fbce4163480 2024-05-09T05:01:54.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.512+0000 7fbccbfff640 1 -- 172.21.15.60:0/992488383 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+20557 (secure 0 0 0) 0x7fbce406d410 con 0x7fbce4163480 2024-05-09T05:01:54.513 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:54.513 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":57,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:01:52.935919+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:01:36.910324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T05:01:38.185514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data_extra1","create_time":"2024-05-09T05:01:47.082451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data_extra2","create_time":"2024-05-09T05:01:50.290433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:01:54.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.514+0000 7fbce9c93640 1 -- 172.21.15.60:0/992488383 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbcd00762d0 msgr2=0x7fbcd0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:54.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.514+0000 7fbce9c93640 1 --2- 172.21.15.60:0/992488383 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbcd00762d0 0x7fbcd0078790 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4007f50 tx=0x7fbcd4007ee0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.514+0000 7fbce9c93640 1 -- 172.21.15.60:0/992488383 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbce4163480 msgr2=0x7fbce41f4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:54.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.514+0000 7fbce9c93640 1 --2- 172.21.15.60:0/992488383 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbce4163480 0x7fbce41f4790 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc009eb0 tx=0x7fbcdc004a40 comp rx=0 tx=0).stop 2024-05-09T05:01:54.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.514+0000 7fbce9c93640 1 -- 172.21.15.60:0/992488383 shutdown_connections 2024-05-09T05:01:54.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.514+0000 7fbce9c93640 1 --2- 172.21.15.60:0/992488383 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbcd00762d0 0x7fbcd0078790 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.514+0000 7fbce9c93640 1 --2- 172.21.15.60:0/992488383 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbce4163480 0x7fbce41f4790 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.514+0000 7fbce9c93640 1 --2- 172.21.15.60:0/992488383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbce40b34b0 0x7fbce41f4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.514+0000 7fbce9c93640 1 --2- 172.21.15.60:0/992488383 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbce40b1700 0x7fbce406ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.514+0000 7fbce9c93640 1 -- 172.21.15.60:0/992488383 >> 172.21.15.60:0/992488383 conn(0x7fbce40b42e0 msgr2=0x7fbce409b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:54.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.514+0000 7fbce9c93640 1 -- 172.21.15.60:0/992488383 shutdown_connections 2024-05-09T05:01:54.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.514+0000 7fbce9c93640 1 -- 172.21.15.60:0/992488383 wait complete. 2024-05-09T05:01:54.526 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T05:01:54.526 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:54.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.629+0000 7f2f04e54640 1 Processor -- start 2024-05-09T05:01:54.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.629+0000 7f2f04e54640 1 -- start start 2024-05-09T05:01:54.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.630+0000 7f2f04e54640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f2f00098af0 0x7f2f0009b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.630+0000 7f2f04e54640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2f000d0070 0x7f2f000b0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.630+0000 7f2f04e54640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f000d05e0 con 0x7f2f00098ed0 2024-05-09T05:01:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.630+0000 7f2f04e54640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f2f000ae9b0 con 0x7f2f000d0070 2024-05-09T05:01:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.630+0000 7f2f04e54640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f2f000aec90 con 0x7f2f00098af0 2024-05-09T05:01:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.630+0000 7f2efed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2f000d0070 0x7f2f000b0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.630+0000 7f2efed9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2f000d0070 0x7f2f000b0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35740/0 (socket says 172.21.15.60:35740) 2024-05-09T05:01:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.630+0000 7f2efed9d640 1 -- 172.21.15.60:0/742715377 learned_addr learned my addr 172.21.15.60:0/742715377 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.630+0000 7f2efe59c640 1 --2- 172.21.15.60:0/742715377 >> v2:172.21.15.114:3300/0 conn(0x7f2f00098af0 0x7f2f0009b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.630+0000 7f2efed9d640 1 -- 172.21.15.60:0/742715377 >> v2:172.21.15.114:3300/0 conn(0x7f2f00098af0 msgr2=0x7f2f0009b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.630+0000 7f2efed9d640 1 --2- 172.21.15.60:0/742715377 >> v2:172.21.15.114:3300/0 conn(0x7f2f00098af0 0x7f2f0009b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.630+0000 7f2efed9d640 1 -- 172.21.15.60:0/742715377 >> v1:172.21.15.114:6789/0 conn(0x7f2f00098ed0 legacy=0x7f2f000b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.631+0000 7f2efed9d640 1 -- 172.21.15.60:0/742715377 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f000aef70 con 0x7f2f000d0070 2024-05-09T05:01:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.631+0000 7f2efed9d640 1 --2- 172.21.15.60:0/742715377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f000d0070 0x7f2f000b0580 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2f0006a120 tx=0x7f2ef802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b988c1389c7dcb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:54.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.632+0000 7f2efd59a640 1 -- 172.21.15.60:0/742715377 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8036070 con 0x7f2f000d0070 2024-05-09T05:01:54.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.632+0000 7f2efd59a640 1 -- 172.21.15.60:0/742715377 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ef8031600 con 0x7f2f000d0070 2024-05-09T05:01:54.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.632+0000 7f2efd59a640 1 -- 172.21.15.60:0/742715377 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef80319d0 con 0x7f2f000d0070 2024-05-09T05:01:54.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.632+0000 7f2f04e54640 1 -- 172.21.15.60:0/742715377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f000d0070 msgr2=0x7f2f000b0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:54.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.632+0000 7f2f04e54640 1 --2- 172.21.15.60:0/742715377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f000d0070 0x7f2f000b0580 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2f0006a120 tx=0x7f2ef802a6a0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.632+0000 7f2f04e54640 1 -- 172.21.15.60:0/742715377 shutdown_connections 2024-05-09T05:01:54.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.632+0000 7f2f04e54640 1 --2- 172.21.15.60:0/742715377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f000d0070 0x7f2f000b0580 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.632+0000 7f2f04e54640 1 --2- 172.21.15.60:0/742715377 >> v2:172.21.15.114:3300/0 conn(0x7f2f00098af0 0x7f2f0009b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.632+0000 7f2f04e54640 1 -- 172.21.15.60:0/742715377 >> 172.21.15.60:0/742715377 conn(0x7f2f000b42e0 msgr2=0x7f2f000b46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:54.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.632+0000 7f2f04e54640 1 -- 172.21.15.60:0/742715377 shutdown_connections 2024-05-09T05:01:54.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.632+0000 7f2f04e54640 1 -- 172.21.15.60:0/742715377 wait complete. 2024-05-09T05:01:54.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.634+0000 7f2f04e54640 1 Processor -- start 2024-05-09T05:01:54.635 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.634+0000 7f2f04e54640 1 -- start start 2024-05-09T05:01:54.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.634+0000 7f2f04e54640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f00098af0 0x7f2f000add10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:54.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.634+0000 7f2f04e54640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f00098ed0 0x7f2f00150e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:54.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.634+0000 7f2f04e54640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f000d0070 0x7f2f00151350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:54.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.634+0000 7f2f04e54640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2f000b1960 con 0x7f2f000d0070 2024-05-09T05:01:54.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.634+0000 7f2f04e54640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2f000b17e0 con 0x7f2f00098af0 2024-05-09T05:01:54.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.634+0000 7f2f04e54640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f2f0009b8e0 con 0x7f2f00098ed0 2024-05-09T05:01:54.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.635+0000 7f2efed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f000d0070 0x7f2f00151350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:54.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.635+0000 7f2efed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f000d0070 0x7f2f00151350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35752/0 (socket says 172.21.15.60:35752) 2024-05-09T05:01:54.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.635+0000 7f2efed9d640 1 -- 172.21.15.60:0/4255636680 learned_addr learned my addr 172.21.15.60:0/4255636680 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:54.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.635+0000 7f2efe59c640 1 --2- 172.21.15.60:0/4255636680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f00098af0 0x7f2f000add10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:54.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.635+0000 7f2efdd9b640 1 --2- 172.21.15.60:0/4255636680 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f00098ed0 0x7f2f00150e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:54.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.635+0000 7f2efed9d640 1 -- 172.21.15.60:0/4255636680 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f00098ed0 msgr2=0x7f2f00150e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:54.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.635+0000 7f2efed9d640 1 --2- 172.21.15.60:0/4255636680 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f00098ed0 0x7f2f00150e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.635+0000 7f2efed9d640 1 -- 172.21.15.60:0/4255636680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f00098af0 msgr2=0x7f2f000add10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:54.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.635+0000 7f2efed9d640 1 --2- 172.21.15.60:0/4255636680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f00098af0 0x7f2f000add10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.635+0000 7f2efed9d640 1 -- 172.21.15.60:0/4255636680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f0014fe50 con 0x7f2f000d0070 2024-05-09T05:01:54.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.635+0000 7f2efed9d640 1 --2- 172.21.15.60:0/4255636680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f000d0070 0x7f2f00151350 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8009f40 tx=0x7f2ef8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:54.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.635+0000 7f2eeeffd640 1 -- 172.21.15.60:0/4255636680 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8036050 con 0x7f2f000d0070 2024-05-09T05:01:54.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.635+0000 7f2eeeffd640 1 -- 172.21.15.60:0/4255636680 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ef8042070 con 0x7f2f000d0070 2024-05-09T05:01:54.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.635+0000 7f2f04e54640 1 -- 172.21.15.60:0/4255636680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f00150130 con 0x7f2f000d0070 2024-05-09T05:01:54.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.635+0000 7f2eeeffd640 1 -- 172.21.15.60:0/4255636680 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8009510 con 0x7f2f000d0070 2024-05-09T05:01:54.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.635+0000 7f2f04e54640 1 -- 172.21.15.60:0/4255636680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f001506f0 con 0x7f2f000d0070 2024-05-09T05:01:54.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.636+0000 7f2f04e54640 1 -- 172.21.15.60:0/4255636680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f00051040 con 0x7f2f000d0070 2024-05-09T05:01:54.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.637+0000 7f2eeeffd640 1 -- 172.21.15.60:0/4255636680 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2ef8007bd0 con 0x7f2f000d0070 2024-05-09T05:01:54.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.638+0000 7f2eeeffd640 1 --2- 172.21.15.60:0/4255636680 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2ee00762d0 0x7f2ee0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:54.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.638+0000 7f2eeeffd640 1 -- 172.21.15.60:0/4255636680 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6911+0+0 (secure 0 0 0) 0x7f2ef80b88e0 con 0x7f2f000d0070 2024-05-09T05:01:54.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.638+0000 7f2efe59c640 1 --2- 172.21.15.60:0/4255636680 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2ee00762d0 0x7f2ee0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:54.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.638+0000 7f2efe59c640 1 --2- 172.21.15.60:0/4255636680 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2ee00762d0 0x7f2ee0078790 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2ef000aa20 tx=0x7f2ef0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:54.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.642+0000 7f2eeeffd640 1 -- 172.21.15.60:0/4255636680 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2ef8081740 con 0x7f2f000d0070 2024-05-09T05:01:54.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.874+0000 7f2f04e54640 1 -- 172.21.15.60:0/4255636680 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f0009ab20 con 0x7f2f000d0070 2024-05-09T05:01:54.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.875+0000 7f2eeeffd640 1 -- 172.21.15.60:0/4255636680 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+5193 (secure 0 0 0) 0x7f2ef8085a40 con 0x7f2f000d0070 2024-05-09T05:01:54.876 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:54.876 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":29,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3604145570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3604145570},{"type":"v1","addr":"172.21.15.60:6834","nonce":3604145570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3949194455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3949194455},{"type":"v1","addr":"172.21.15.114:6836","nonce":3949194455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3288630382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3288630382},{"type":"v1","addr":"172.21.15.114:6837","nonce":3288630382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:39.237498+0000","modified":"2024-05-09T05:01:51.305520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5415":{"gid":5415,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2425263408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2425263408},{"type":"v1","addr":"172.21.15.60:6835","nonce":2425263408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9,10,11],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5415,"qdb_cluster":[5415]},"id":3}]} 2024-05-09T05:01:54.876 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 29 2024-05-09T05:01:54.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.877+0000 7f2f04e54640 1 -- 172.21.15.60:0/4255636680 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2ee00762d0 msgr2=0x7f2ee0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:54.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.877+0000 7f2f04e54640 1 --2- 172.21.15.60:0/4255636680 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2ee00762d0 0x7f2ee0078790 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2ef000aa20 tx=0x7f2ef0008040 comp rx=0 tx=0).stop 2024-05-09T05:01:54.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.877+0000 7f2f04e54640 1 -- 172.21.15.60:0/4255636680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f000d0070 msgr2=0x7f2f00151350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:54.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.877+0000 7f2f04e54640 1 --2- 172.21.15.60:0/4255636680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f000d0070 0x7f2f00151350 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8009f40 tx=0x7f2ef8004a40 comp rx=0 tx=0).stop 2024-05-09T05:01:54.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.877+0000 7f2f04e54640 1 -- 172.21.15.60:0/4255636680 shutdown_connections 2024-05-09T05:01:54.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.877+0000 7f2f04e54640 1 --2- 172.21.15.60:0/4255636680 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2ee00762d0 0x7f2ee0078790 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.877+0000 7f2f04e54640 1 --2- 172.21.15.60:0/4255636680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f000d0070 0x7f2f00151350 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.877+0000 7f2f04e54640 1 --2- 172.21.15.60:0/4255636680 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f00098ed0 0x7f2f00150e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.877+0000 7f2f04e54640 1 --2- 172.21.15.60:0/4255636680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f00098af0 0x7f2f000add10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.877+0000 7f2f04e54640 1 -- 172.21.15.60:0/4255636680 >> 172.21.15.60:0/4255636680 conn(0x7f2f000b42e0 msgr2=0x7f2f0009bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:54.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.877+0000 7f2f04e54640 1 -- 172.21.15.60:0/4255636680 shutdown_connections 2024-05-09T05:01:54.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.877+0000 7f2f04e54640 1 -- 172.21.15.60:0/4255636680 wait complete. 2024-05-09T05:01:54.889 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:01:54.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.993+0000 7f78b430a640 1 Processor -- start 2024-05-09T05:01:54.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.993+0000 7f78b430a640 1 -- start start 2024-05-09T05:01:54.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.994+0000 7f78b430a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f78ac0b1040 0x7f78ac0b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:54.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.994+0000 7f78b430a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f78ac0b2580 0x7f78ac0b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:54.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.994+0000 7f78b430a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78ac09d890 con 0x7f78ac096d70 2024-05-09T05:01:54.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.994+0000 7f78b430a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f78ac097150 con 0x7f78ac0b2580 2024-05-09T05:01:54.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.994+0000 7f78b430a640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f78ac09c020 con 0x7f78ac0b1040 2024-05-09T05:01:54.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.994+0000 7f78b28a7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f78ac0b2580 0x7f78ac0b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:54.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.994+0000 7f78b28a7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f78ac0b2580 0x7f78ac0b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35754/0 (socket says 172.21.15.60:35754) 2024-05-09T05:01:54.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.994+0000 7f78b28a7640 1 -- 172.21.15.60:0/2294463560 learned_addr learned my addr 172.21.15.60:0/2294463560 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:54.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.994+0000 7f78b18a5640 1 --2- 172.21.15.60:0/2294463560 >> v2:172.21.15.114:3300/0 conn(0x7f78ac0b1040 0x7f78ac0b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:54.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.994+0000 7f78b28a7640 1 -- 172.21.15.60:0/2294463560 >> v2:172.21.15.114:3300/0 conn(0x7f78ac0b1040 msgr2=0x7f78ac0b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:54.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.994+0000 7f78b28a7640 1 --2- 172.21.15.60:0/2294463560 >> v2:172.21.15.114:3300/0 conn(0x7f78ac0b1040 0x7f78ac0b14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.994+0000 7f78b28a7640 1 -- 172.21.15.60:0/2294463560 >> v1:172.21.15.114:6789/0 conn(0x7f78ac096d70 legacy=0x7f78ac09d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:54.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.994+0000 7f78b28a7640 1 -- 172.21.15.60:0/2294463560 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ac09c300 con 0x7f78ac0b2580 2024-05-09T05:01:54.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.995+0000 7f78b28a7640 1 --2- 172.21.15.60:0/2294463560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78ac0b2580 0x7f78ac0b29c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0743f0 tx=0x7f78a402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34bbf829ee53f388 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:54.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.995+0000 7f78b10a4640 1 -- 172.21.15.60:0/2294463560 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a4036070 con 0x7f78ac0b2580 2024-05-09T05:01:54.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.995+0000 7f78b10a4640 1 -- 172.21.15.60:0/2294463560 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78a4031600 con 0x7f78ac0b2580 2024-05-09T05:01:54.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.995+0000 7f78b10a4640 1 -- 172.21.15.60:0/2294463560 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a40319d0 con 0x7f78ac0b2580 2024-05-09T05:01:54.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.996+0000 7f78b430a640 1 -- 172.21.15.60:0/2294463560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78ac0b2580 msgr2=0x7f78ac0b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:54.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.996+0000 7f78b430a640 1 --2- 172.21.15.60:0/2294463560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78ac0b2580 0x7f78ac0b29c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0743f0 tx=0x7f78a402a6a0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.996+0000 7f78b430a640 1 -- 172.21.15.60:0/2294463560 shutdown_connections 2024-05-09T05:01:54.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.996+0000 7f78b430a640 1 --2- 172.21.15.60:0/2294463560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78ac0b2580 0x7f78ac0b29c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.996+0000 7f78b430a640 1 --2- 172.21.15.60:0/2294463560 >> v2:172.21.15.114:3300/0 conn(0x7f78ac0b1040 0x7f78ac0b14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:54.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.996+0000 7f78b430a640 1 -- 172.21.15.60:0/2294463560 >> 172.21.15.60:0/2294463560 conn(0x7f78ac072390 msgr2=0x7f78ac0727a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:54.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.996+0000 7f78b430a640 1 -- 172.21.15.60:0/2294463560 shutdown_connections 2024-05-09T05:01:54.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.996+0000 7f78b430a640 1 -- 172.21.15.60:0/2294463560 wait complete. 2024-05-09T05:01:54.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.997+0000 7f78b430a640 1 Processor -- start 2024-05-09T05:01:54.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.997+0000 7f78b430a640 1 -- start start 2024-05-09T05:01:54.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78b430a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78ac096d70 0x7f78ac0aa620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:54.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78b430a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78ac0b1040 0x7f78ac1524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:54.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78b430a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78ac0b2580 0x7f78ac091cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:55.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78b430a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac09cba0 con 0x7f78ac096d70 2024-05-09T05:01:55.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78b430a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac09c8c0 con 0x7f78ac0b1040 2024-05-09T05:01:55.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78b430a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f78ac0b19f0 con 0x7f78ac0b2580 2024-05-09T05:01:55.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78b20a6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78ac096d70 0x7f78ac0aa620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:55.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78b20a6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78ac096d70 0x7f78ac0aa620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35768/0 (socket says 172.21.15.60:35768) 2024-05-09T05:01:55.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78b20a6640 1 -- 172.21.15.60:0/16079407 learned_addr learned my addr 172.21.15.60:0/16079407 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:55.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78b18a5640 1 --2- 172.21.15.60:0/16079407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78ac0b1040 0x7f78ac1524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:55.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78b20a6640 1 -- 172.21.15.60:0/16079407 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78ac0b2580 msgr2=0x7f78ac091cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:55.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78b20a6640 1 --2- 172.21.15.60:0/16079407 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78ac0b2580 0x7f78ac091cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78b20a6640 1 -- 172.21.15.60:0/16079407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78ac0b1040 msgr2=0x7f78ac1524d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:55.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78b20a6640 1 --2- 172.21.15.60:0/16079407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78ac0b1040 0x7f78ac1524d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78b20a6640 1 -- 172.21.15.60:0/16079407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ac0aab60 con 0x7f78ac096d70 2024-05-09T05:01:55.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78b20a6640 1 --2- 172.21.15.60:0/16079407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78ac096d70 0x7f78ac0aa620 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f78a800c460 tx=0x7f78a800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:55.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78a2ffd640 1 -- 172.21.15.60:0/16079407 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a8016020 con 0x7f78ac096d70 2024-05-09T05:01:55.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78a2ffd640 1 -- 172.21.15.60:0/16079407 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78a8012870 con 0x7f78ac096d70 2024-05-09T05:01:55.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78a2ffd640 1 -- 172.21.15.60:0/16079407 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a8012c60 con 0x7f78ac096d70 2024-05-09T05:01:55.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78b430a640 1 -- 172.21.15.60:0/16079407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78ac152e40 con 0x7f78ac096d70 2024-05-09T05:01:55.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.998+0000 7f78b430a640 1 -- 172.21.15.60:0/16079407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78ac0914f0 con 0x7f78ac096d70 2024-05-09T05:01:55.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:54.999+0000 7f78b430a640 1 -- 172.21.15.60:0/16079407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78ac0b1c00 con 0x7f78ac096d70 2024-05-09T05:01:55.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.000+0000 7f78a2ffd640 1 -- 172.21.15.60:0/16079407 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f78a8019020 con 0x7f78ac096d70 2024-05-09T05:01:55.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.001+0000 7f78a2ffd640 1 --2- 172.21.15.60:0/16079407 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f78900762d0 0x7f7890078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:55.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.001+0000 7f78a2ffd640 1 -- 172.21.15.60:0/16079407 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6911+0+0 (secure 0 0 0) 0x7f78a8097490 con 0x7f78ac096d70 2024-05-09T05:01:55.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.001+0000 7f78b18a5640 1 --2- 172.21.15.60:0/16079407 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f78900762d0 0x7f7890078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:55.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.001+0000 7f78b18a5640 1 --2- 172.21.15.60:0/16079407 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f78900762d0 0x7f7890078790 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f78ac09d410 tx=0x7f789c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:55.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.005+0000 7f78a2ffd640 1 -- 172.21.15.60:0/16079407 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f78ac0b1c00 con 0x7f78ac096d70 2024-05-09T05:01:55.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.235+0000 7f78b430a640 1 -- 172.21.15.60:0/16079407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78ac051040 con 0x7f78ac096d70 2024-05-09T05:01:55.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.235+0000 7f78a2ffd640 1 -- 172.21.15.60:0/16079407 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+5193 (secure 0 0 0) 0x7f78a8020070 con 0x7f78ac096d70 2024-05-09T05:01:55.236 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:55.236 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":29,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3604145570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3604145570},{"type":"v1","addr":"172.21.15.60:6834","nonce":3604145570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3949194455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3949194455},{"type":"v1","addr":"172.21.15.114:6836","nonce":3949194455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3288630382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3288630382},{"type":"v1","addr":"172.21.15.114:6837","nonce":3288630382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:01:39.237498+0000","modified":"2024-05-09T05:01:51.305520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5415":{"gid":5415,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6835/2425263408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":2425263408},{"type":"v1","addr":"172.21.15.60:6835","nonce":2425263408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9,10,11],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5415,"qdb_cluster":[5415]},"id":3}]} 2024-05-09T05:01:55.236 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 29 2024-05-09T05:01:55.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.237+0000 7f78b430a640 1 -- 172.21.15.60:0/16079407 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f78900762d0 msgr2=0x7f7890078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:55.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.237+0000 7f78b430a640 1 --2- 172.21.15.60:0/16079407 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f78900762d0 0x7f7890078790 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f78ac09d410 tx=0x7f789c008040 comp rx=0 tx=0).stop 2024-05-09T05:01:55.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.237+0000 7f78b430a640 1 -- 172.21.15.60:0/16079407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78ac096d70 msgr2=0x7f78ac0aa620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:55.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.237+0000 7f78b430a640 1 --2- 172.21.15.60:0/16079407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78ac096d70 0x7f78ac0aa620 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f78a800c460 tx=0x7f78a800c920 comp rx=0 tx=0).stop 2024-05-09T05:01:55.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.237+0000 7f78b430a640 1 -- 172.21.15.60:0/16079407 shutdown_connections 2024-05-09T05:01:55.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.237+0000 7f78b430a640 1 --2- 172.21.15.60:0/16079407 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f78900762d0 0x7f7890078790 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.237+0000 7f78b430a640 1 --2- 172.21.15.60:0/16079407 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f78ac0b2580 0x7f78ac091cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.237+0000 7f78b430a640 1 --2- 172.21.15.60:0/16079407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78ac0b1040 0x7f78ac1524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.237+0000 7f78b430a640 1 --2- 172.21.15.60:0/16079407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78ac096d70 0x7f78ac0aa620 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.238+0000 7f78b430a640 1 -- 172.21.15.60:0/16079407 >> 172.21.15.60:0/16079407 conn(0x7f78ac072390 msgr2=0x7f78ac0af2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:55.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.238+0000 7f78b430a640 1 -- 172.21.15.60:0/16079407 shutdown_connections 2024-05-09T05:01:55.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.238+0000 7f78b430a640 1 -- 172.21.15.60:0/16079407 wait complete. 2024-05-09T05:01:55.249 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:01:55.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.352+0000 7fb6dae7e640 1 Processor -- start 2024-05-09T05:01:55.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.352+0000 7fb6dae7e640 1 -- start start 2024-05-09T05:01:55.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.353+0000 7fb6dae7e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb6d40ae1f0 0x7fb6d409d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.353+0000 7fb6dae7e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d40b1040 0x7fb6d40b14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.353+0000 7fb6dae7e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d40b3db0 con 0x7fb6d40aea30 2024-05-09T05:01:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.353+0000 7fb6dae7e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fb6d409cdb0 con 0x7fb6d40ae1f0 2024-05-09T05:01:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.353+0000 7fb6dae7e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb6d409c7f0 con 0x7fb6d40b1040 2024-05-09T05:01:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.353+0000 7fb6d8c1a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb6d40ae1f0 0x7fb6d409d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.353+0000 7fb6d8c1a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb6d40ae1f0 0x7fb6d409d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35776/0 (socket says 172.21.15.60:35776) 2024-05-09T05:01:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.353+0000 7fb6d941b640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb6d40aea30 0x7fb6d40aeec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57186/0 (socket says 172.21.15.60:57186) 2024-05-09T05:01:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.353+0000 7fb6d8c1a640 1 -- 172.21.15.60:0/2583814154 learned_addr learned my addr 172.21.15.60:0/2583814154 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.353+0000 7fb6d3fff640 1 --2- 172.21.15.60:0/2583814154 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d40b1040 0x7fb6d40b14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.353+0000 7fb6d8c1a640 1 -- 172.21.15.60:0/2583814154 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d40b1040 msgr2=0x7fb6d40b14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.353+0000 7fb6d8c1a640 1 --2- 172.21.15.60:0/2583814154 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d40b1040 0x7fb6d40b14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.353+0000 7fb6d8c1a640 1 -- 172.21.15.60:0/2583814154 >> v1:172.21.15.60:6789/0 conn(0x7fb6d40aea30 legacy=0x7fb6d40aeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:55.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.353+0000 7fb6d37fe640 1 -- 172.21.15.60:0/2583814154 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1208494167 0 0) 0x7fb6d40b3db0 con 0x7fb6d40aea30 2024-05-09T05:01:55.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.353+0000 7fb6d8c1a640 1 -- 172.21.15.60:0/2583814154 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d409cad0 con 0x7fb6d40ae1f0 2024-05-09T05:01:55.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.353+0000 7fb6d3fff640 1 --2- 172.21.15.60:0/2583814154 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d40b1040 0x7fb6d40b14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:55.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.354+0000 7fb6d8c1a640 1 --2- 172.21.15.60:0/2583814154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d40ae1f0 0x7fb6d409d180 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8009870 tx=0x7fb6c802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad6a45d10274e07c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:55.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.354+0000 7fb6d37fe640 1 -- 172.21.15.60:0/2583814154 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8037070 con 0x7fb6d40ae1f0 2024-05-09T05:01:55.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.354+0000 7fb6d37fe640 1 -- 172.21.15.60:0/2583814154 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6c8032440 con 0x7fb6d40ae1f0 2024-05-09T05:01:55.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.354+0000 7fb6d37fe640 1 -- 172.21.15.60:0/2583814154 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8032810 con 0x7fb6d40ae1f0 2024-05-09T05:01:55.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.355+0000 7fb6dae7e640 1 -- 172.21.15.60:0/2583814154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d40ae1f0 msgr2=0x7fb6d409d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:55.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.355+0000 7fb6dae7e640 1 --2- 172.21.15.60:0/2583814154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d40ae1f0 0x7fb6d409d180 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8009870 tx=0x7fb6c802a1c0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.355+0000 7fb6dae7e640 1 -- 172.21.15.60:0/2583814154 shutdown_connections 2024-05-09T05:01:55.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.355+0000 7fb6dae7e640 1 --2- 172.21.15.60:0/2583814154 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d40b1040 0x7fb6d40b14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.355+0000 7fb6dae7e640 1 --2- 172.21.15.60:0/2583814154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d40ae1f0 0x7fb6d409d180 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.355+0000 7fb6dae7e640 1 -- 172.21.15.60:0/2583814154 >> 172.21.15.60:0/2583814154 conn(0x7fb6d4072390 msgr2=0x7fb6d40727a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:55.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.355+0000 7fb6dae7e640 1 -- 172.21.15.60:0/2583814154 shutdown_connections 2024-05-09T05:01:55.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.355+0000 7fb6dae7e640 1 -- 172.21.15.60:0/2583814154 wait complete. 2024-05-09T05:01:55.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.356+0000 7fb6dae7e640 1 Processor -- start 2024-05-09T05:01:55.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.356+0000 7fb6dae7e640 1 -- start start 2024-05-09T05:01:55.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.357+0000 7fb6dae7e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d40ae1f0 0x7fb6d4155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:55.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.357+0000 7fb6dae7e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6d40aea30 0x7fb6d4157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:55.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.357+0000 7fb6dae7e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d40b1040 0x7fb6d4158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:55.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.357+0000 7fb6dae7e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d40b19e0 con 0x7fb6d40ae1f0 2024-05-09T05:01:55.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.357+0000 7fb6dae7e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d40b2880 con 0x7fb6d40aea30 2024-05-09T05:01:55.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.357+0000 7fb6dae7e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fb6d40ab4b0 con 0x7fb6d40b1040 2024-05-09T05:01:55.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.357+0000 7fb6d8c1a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d40ae1f0 0x7fb6d4155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:55.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.357+0000 7fb6d8c1a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d40ae1f0 0x7fb6d4155550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35788/0 (socket says 172.21.15.60:35788) 2024-05-09T05:01:55.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.357+0000 7fb6d8c1a640 1 -- 172.21.15.60:0/4276020851 learned_addr learned my addr 172.21.15.60:0/4276020851 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:55.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.357+0000 7fb6d941b640 1 --2- 172.21.15.60:0/4276020851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d40b1040 0x7fb6d4158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:55.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.357+0000 7fb6d3fff640 1 --2- 172.21.15.60:0/4276020851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6d40aea30 0x7fb6d4157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:55.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.357+0000 7fb6d8c1a640 1 -- 172.21.15.60:0/4276020851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d40b1040 msgr2=0x7fb6d4158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:55.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.357+0000 7fb6d8c1a640 1 --2- 172.21.15.60:0/4276020851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d40b1040 0x7fb6d4158310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.357+0000 7fb6d8c1a640 1 -- 172.21.15.60:0/4276020851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6d40aea30 msgr2=0x7fb6d4157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:55.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.357+0000 7fb6d8c1a640 1 --2- 172.21.15.60:0/4276020851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6d40aea30 0x7fb6d4157dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.357+0000 7fb6d8c1a640 1 -- 172.21.15.60:0/4276020851 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d4162250 con 0x7fb6d40ae1f0 2024-05-09T05:01:55.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.357+0000 7fb6d3fff640 1 --2- 172.21.15.60:0/4276020851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6d40aea30 0x7fb6d4157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:55.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.357+0000 7fb6d8c1a640 1 --2- 172.21.15.60:0/4276020851 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d40ae1f0 0x7fb6d4155550 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8000c00 tx=0x7fb6c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:55.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.358+0000 7fb6d17fa640 1 -- 172.21.15.60:0/4276020851 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8041020 con 0x7fb6d40ae1f0 2024-05-09T05:01:55.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.358+0000 7fb6d17fa640 1 -- 172.21.15.60:0/4276020851 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6c80306c0 con 0x7fb6d40ae1f0 2024-05-09T05:01:55.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.358+0000 7fb6dae7e640 1 -- 172.21.15.60:0/4276020851 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6d4162530 con 0x7fb6d40ae1f0 2024-05-09T05:01:55.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.358+0000 7fb6d17fa640 1 -- 172.21.15.60:0/4276020851 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8037040 con 0x7fb6d40ae1f0 2024-05-09T05:01:55.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.358+0000 7fb6dae7e640 1 -- 172.21.15.60:0/4276020851 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6d4159c30 con 0x7fb6d40ae1f0 2024-05-09T05:01:55.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.359+0000 7fb6dae7e640 1 -- 172.21.15.60:0/4276020851 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6d4075b10 con 0x7fb6d40ae1f0 2024-05-09T05:01:55.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.359+0000 7fb6d17fa640 1 -- 172.21.15.60:0/4276020851 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb6c8044020 con 0x7fb6d40ae1f0 2024-05-09T05:01:55.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.360+0000 7fb6d17fa640 1 --2- 172.21.15.60:0/4276020851 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb6b80762d0 0x7fb6b8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:55.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.360+0000 7fb6d17fa640 1 -- 172.21.15.60:0/4276020851 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6911+0+0 (secure 0 0 0) 0x7fb6c80b8300 con 0x7fb6d40ae1f0 2024-05-09T05:01:55.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.360+0000 7fb6d3fff640 1 --2- 172.21.15.60:0/4276020851 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb6b80762d0 0x7fb6b8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:55.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.361+0000 7fb6d3fff640 1 --2- 172.21.15.60:0/4276020851 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb6b80762d0 0x7fb6b8078790 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0006fd0 tx=0x7fb6c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:55.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.364+0000 7fb6d17fa640 1 -- 172.21.15.60:0/4276020851 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb6d4075b10 con 0x7fb6d40ae1f0 2024-05-09T05:01:55.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.554+0000 7fb6dae7e640 1 -- 172.21.15.60:0/4276020851 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb6d4051040 con 0x7fb6d40ae1f0 2024-05-09T05:01:55.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.555+0000 7fb6d17fa640 1 -- 172.21.15.60:0/4276020851 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+20557 (secure 0 0 0) 0x7fb6c804b030 con 0x7fb6d40ae1f0 2024-05-09T05:01:55.556 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:01:55.556 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":57,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:01:52.935919+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:01:36.910324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T05:01:38.185514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data_extra1","create_time":"2024-05-09T05:01:47.082451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data_extra2","create_time":"2024-05-09T05:01:50.290433+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:01:55.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.557+0000 7fb6dae7e640 1 -- 172.21.15.60:0/4276020851 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb6b80762d0 msgr2=0x7fb6b8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:55.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.557+0000 7fb6dae7e640 1 --2- 172.21.15.60:0/4276020851 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb6b80762d0 0x7fb6b8078790 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0006fd0 tx=0x7fb6c0008040 comp rx=0 tx=0).stop 2024-05-09T05:01:55.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.557+0000 7fb6dae7e640 1 -- 172.21.15.60:0/4276020851 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d40ae1f0 msgr2=0x7fb6d4155550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:55.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.557+0000 7fb6dae7e640 1 --2- 172.21.15.60:0/4276020851 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d40ae1f0 0x7fb6d4155550 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8000c00 tx=0x7fb6c80046c0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.558+0000 7fb6dae7e640 1 -- 172.21.15.60:0/4276020851 shutdown_connections 2024-05-09T05:01:55.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.558+0000 7fb6dae7e640 1 --2- 172.21.15.60:0/4276020851 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fb6b80762d0 0x7fb6b8078790 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.558+0000 7fb6dae7e640 1 --2- 172.21.15.60:0/4276020851 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fb6d40b1040 0x7fb6d4158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.558+0000 7fb6dae7e640 1 --2- 172.21.15.60:0/4276020851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6d40aea30 0x7fb6d4157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.558+0000 7fb6dae7e640 1 --2- 172.21.15.60:0/4276020851 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6d40ae1f0 0x7fb6d4155550 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.558+0000 7fb6dae7e640 1 -- 172.21.15.60:0/4276020851 >> 172.21.15.60:0/4276020851 conn(0x7fb6d4072390 msgr2=0x7fb6d40ad780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:55.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.558+0000 7fb6dae7e640 1 -- 172.21.15.60:0/4276020851 shutdown_connections 2024-05-09T05:01:55.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.558+0000 7fb6dae7e640 1 -- 172.21.15.60:0/4276020851 wait complete. 2024-05-09T05:01:55.569 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T05:01:55.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.673+0000 7f9b70220640 1 Processor -- start 2024-05-09T05:01:55.674 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.674+0000 7f9b70220640 1 -- start start 2024-05-09T05:01:55.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.674+0000 7f9b70220640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9b6806e7f0 0x7f9b680d2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.674+0000 7f9b70220640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9b680d2650 0x7f9b680d1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.674+0000 7f9b70220640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b680b31d0 con 0x7f9b6806fcb0 2024-05-09T05:01:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.674+0000 7f9b70220640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f9b680b2f80 con 0x7f9b6806e7f0 2024-05-09T05:01:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.674+0000 7f9b70220640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f9b680b34b0 con 0x7f9b680d2650 2024-05-09T05:01:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.675+0000 7f9b6dfbc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9b6806e7f0 0x7f9b680d2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.675+0000 7f9b6dfbc640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9b6806e7f0 0x7f9b680d2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35796/0 (socket says 172.21.15.60:35796) 2024-05-09T05:01:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.675+0000 7f9b6dfbc640 1 -- 172.21.15.60:0/3381213446 learned_addr learned my addr 172.21.15.60:0/3381213446 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.675+0000 7f9b6e7bd640 1 --2- 172.21.15.60:0/3381213446 >> v2:172.21.15.114:3300/0 conn(0x7f9b680d2650 0x7f9b680d1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.675+0000 7f9b6dfbc640 1 -- 172.21.15.60:0/3381213446 >> v2:172.21.15.114:3300/0 conn(0x7f9b680d2650 msgr2=0x7f9b680d1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.675+0000 7f9b6dfbc640 1 --2- 172.21.15.60:0/3381213446 >> v2:172.21.15.114:3300/0 conn(0x7f9b680d2650 0x7f9b680d1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.675+0000 7f9b6dfbc640 1 -- 172.21.15.60:0/3381213446 >> v1:172.21.15.114:6789/0 conn(0x7f9b6806fcb0 legacy=0x7f9b680d1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.675+0000 7f9b6dfbc640 1 -- 172.21.15.60:0/3381213446 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b680b3790 con 0x7f9b6806e7f0 2024-05-09T05:01:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.675+0000 7f9b6e7bd640 1 --2- 172.21.15.60:0/3381213446 >> v2:172.21.15.114:3300/0 conn(0x7f9b680d2650 0x7f9b680d1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:55.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.676+0000 7f9b6dfbc640 1 --2- 172.21.15.60:0/3381213446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b6806e7f0 0x7f9b680d2b50 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9b680708e0 tx=0x7f9b6002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d2829a5daf3795 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:55.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.676+0000 7f9b6cfba640 1 -- 172.21.15.60:0/3381213446 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b60037070 con 0x7f9b6806e7f0 2024-05-09T05:01:55.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.676+0000 7f9b6cfba640 1 -- 172.21.15.60:0/3381213446 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b60032440 con 0x7f9b6806e7f0 2024-05-09T05:01:55.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.676+0000 7f9b6cfba640 1 -- 172.21.15.60:0/3381213446 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b60032810 con 0x7f9b6806e7f0 2024-05-09T05:01:55.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.676+0000 7f9b70220640 1 -- 172.21.15.60:0/3381213446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b6806e7f0 msgr2=0x7f9b680d2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:55.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.676+0000 7f9b70220640 1 --2- 172.21.15.60:0/3381213446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b6806e7f0 0x7f9b680d2b50 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9b680708e0 tx=0x7f9b6002a140 comp rx=0 tx=0).stop 2024-05-09T05:01:55.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.676+0000 7f9b70220640 1 -- 172.21.15.60:0/3381213446 shutdown_connections 2024-05-09T05:01:55.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.676+0000 7f9b70220640 1 --2- 172.21.15.60:0/3381213446 >> v2:172.21.15.114:3300/0 conn(0x7f9b680d2650 0x7f9b680d1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.676+0000 7f9b70220640 1 --2- 172.21.15.60:0/3381213446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b6806e7f0 0x7f9b680d2b50 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.676+0000 7f9b70220640 1 -- 172.21.15.60:0/3381213446 >> 172.21.15.60:0/3381213446 conn(0x7f9b680993f0 msgr2=0x7f9b68099800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:55.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.676+0000 7f9b70220640 1 -- 172.21.15.60:0/3381213446 shutdown_connections 2024-05-09T05:01:55.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.676+0000 7f9b70220640 1 -- 172.21.15.60:0/3381213446 wait complete. 2024-05-09T05:01:55.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.678+0000 7f9b70220640 1 Processor -- start 2024-05-09T05:01:55.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.678+0000 7f9b70220640 1 -- start start 2024-05-09T05:01:55.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.678+0000 7f9b70220640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b6806e7f0 0x7f9b6814fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:55.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.678+0000 7f9b70220640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b6806fcb0 0x7f9b680aab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:55.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.678+0000 7f9b70220640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b680d2650 0x7f9b6814e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:55.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.678+0000 7f9b70220640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9b680d1ca0 con 0x7f9b6806e7f0 2024-05-09T05:01:55.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.678+0000 7f9b70220640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9b680b3b00 con 0x7f9b6806fcb0 2024-05-09T05:01:55.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.678+0000 7f9b70220640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9b680d1e20 con 0x7f9b680d2650 2024-05-09T05:01:55.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.678+0000 7f9b6dfbc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b6806e7f0 0x7f9b6814fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:55.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.679+0000 7f9b6dfbc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b6806e7f0 0x7f9b6814fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35798/0 (socket says 172.21.15.60:35798) 2024-05-09T05:01:55.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.679+0000 7f9b6dfbc640 1 -- 172.21.15.60:0/2116927806 learned_addr learned my addr 172.21.15.60:0/2116927806 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:55.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.679+0000 7f9b6e7bd640 1 --2- 172.21.15.60:0/2116927806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b680d2650 0x7f9b6814e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:55.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.679+0000 7f9b6d7bb640 1 --2- 172.21.15.60:0/2116927806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b6806fcb0 0x7f9b680aab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:55.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.679+0000 7f9b6dfbc640 1 -- 172.21.15.60:0/2116927806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b680d2650 msgr2=0x7f9b6814e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:55.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.679+0000 7f9b6dfbc640 1 --2- 172.21.15.60:0/2116927806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b680d2650 0x7f9b6814e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.679+0000 7f9b6dfbc640 1 -- 172.21.15.60:0/2116927806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b6806fcb0 msgr2=0x7f9b680aab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:55.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.679+0000 7f9b6dfbc640 1 --2- 172.21.15.60:0/2116927806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b6806fcb0 0x7f9b680aab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.679+0000 7f9b6dfbc640 1 -- 172.21.15.60:0/2116927806 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b6814d6c0 con 0x7f9b6806e7f0 2024-05-09T05:01:55.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.679+0000 7f9b6dfbc640 1 --2- 172.21.15.60:0/2116927806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b6806e7f0 0x7f9b6814fb10 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f9b60000c00 tx=0x7f9b600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:55.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.679+0000 7f9b5e7fc640 1 -- 172.21.15.60:0/2116927806 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b60041070 con 0x7f9b6806e7f0 2024-05-09T05:01:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.679+0000 7f9b5e7fc640 1 -- 172.21.15.60:0/2116927806 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b6003c870 con 0x7f9b6806e7f0 2024-05-09T05:01:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.679+0000 7f9b5e7fc640 1 -- 172.21.15.60:0/2116927806 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b60037040 con 0x7f9b6806e7f0 2024-05-09T05:01:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.679+0000 7f9b70220640 1 -- 172.21.15.60:0/2116927806 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b6814d9a0 con 0x7f9b6806e7f0 2024-05-09T05:01:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.679+0000 7f9b70220640 1 -- 172.21.15.60:0/2116927806 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b6814df60 con 0x7f9b6806e7f0 2024-05-09T05:01:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.680+0000 7f9b70220640 1 -- 172.21.15.60:0/2116927806 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b68051040 con 0x7f9b6806e7f0 2024-05-09T05:01:55.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.681+0000 7f9b5e7fc640 1 -- 172.21.15.60:0/2116927806 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9b60044070 con 0x7f9b6806e7f0 2024-05-09T05:01:55.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.681+0000 7f9b5e7fc640 1 --2- 172.21.15.60:0/2116927806 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9b4c0762d0 0x7f9b4c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:55.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.681+0000 7f9b5e7fc640 1 -- 172.21.15.60:0/2116927806 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6911+0+0 (secure 0 0 0) 0x7f9b600b8660 con 0x7f9b6806e7f0 2024-05-09T05:01:55.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.682+0000 7f9b6d7bb640 1 --2- 172.21.15.60:0/2116927806 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9b4c0762d0 0x7f9b4c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:55.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.682+0000 7f9b6d7bb640 1 --2- 172.21.15.60:0/2116927806 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9b4c0762d0 0x7f9b4c078790 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9b681f8990 tx=0x7f9b58007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:55.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.686+0000 7f9b5e7fc640 1 -- 172.21.15.60:0/2116927806 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9b60081420 con 0x7f9b6806e7f0 2024-05-09T05:01:55.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.920+0000 7f9b70220640 1 -- 172.21.15.60:0/2116927806 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f9b680998d0 con 0x7f9b6806e7f0 2024-05-09T05:01:55.921 INFO:tasks.ceph.mon.a.smithi060.stderr:2024-05-09T05:01:55.921+0000 7fde6feeb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T05:01:55.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.922+0000 7f9b5e7fc640 1 -- 172.21.15.60:0/2116927806 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v30) ==== 161+0+0 (secure 0 0 0) 0x7f9b60085720 con 0x7f9b6806e7f0 2024-05-09T05:01:55.922 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T05:01:55.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.925+0000 7f9b70220640 1 -- 172.21.15.60:0/2116927806 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9b4c0762d0 msgr2=0x7f9b4c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:55.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.925+0000 7f9b70220640 1 --2- 172.21.15.60:0/2116927806 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9b4c0762d0 0x7f9b4c078790 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9b681f8990 tx=0x7f9b58007e30 comp rx=0 tx=0).stop 2024-05-09T05:01:55.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.925+0000 7f9b70220640 1 -- 172.21.15.60:0/2116927806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b6806e7f0 msgr2=0x7f9b6814fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:55.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.925+0000 7f9b70220640 1 --2- 172.21.15.60:0/2116927806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b6806e7f0 0x7f9b6814fb10 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f9b60000c00 tx=0x7f9b600046c0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.925+0000 7f9b70220640 1 -- 172.21.15.60:0/2116927806 shutdown_connections 2024-05-09T05:01:55.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.926+0000 7f9b70220640 1 --2- 172.21.15.60:0/2116927806 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9b4c0762d0 0x7f9b4c078790 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.926+0000 7f9b70220640 1 --2- 172.21.15.60:0/2116927806 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b680d2650 0x7f9b6814e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.926+0000 7f9b70220640 1 --2- 172.21.15.60:0/2116927806 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b6806fcb0 0x7f9b680aab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.926+0000 7f9b70220640 1 --2- 172.21.15.60:0/2116927806 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b6806e7f0 0x7f9b6814fb10 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:55.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.926+0000 7f9b70220640 1 -- 172.21.15.60:0/2116927806 >> 172.21.15.60:0/2116927806 conn(0x7f9b680993f0 msgr2=0x7f9b6815fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:55.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.926+0000 7f9b70220640 1 -- 172.21.15.60:0/2116927806 shutdown_connections 2024-05-09T05:01:55.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:55.926+0000 7f9b70220640 1 -- 172.21.15.60:0/2116927806 wait complete. 2024-05-09T05:01:55.929 INFO:tasks.ceph.mds.c.smithi060.stderr: -4953> 2024-05-09T05:01:45.839+0000 7f1c4abfe640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T05:01:55.941 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-09T05:01:55.965 INFO:tasks.ceph.mds.c.smithi060.stdout:starting mds.c at 2024-05-09T05:01:56.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.048+0000 7f639d82a640 1 Processor -- start 2024-05-09T05:01:56.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.048+0000 7f639d82a640 1 -- start start 2024-05-09T05:01:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.049+0000 7f639d82a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f63980b1040 0x7f63980b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.049+0000 7f639d82a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f63980b2580 0x7f63980b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.049+0000 7f639d82a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f639809d890 con 0x7f6398096d70 2024-05-09T05:01:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.049+0000 7f639d82a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f6398097150 con 0x7f63980b2580 2024-05-09T05:01:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.049+0000 7f639d82a640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f639809c020 con 0x7f63980b1040 2024-05-09T05:01:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.049+0000 7f63977fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f63980b2580 0x7f63980b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.049+0000 7f63977fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f63980b2580 0x7f63980b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35804/0 (socket says 172.21.15.60:35804) 2024-05-09T05:01:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.049+0000 7f63977fe640 1 -- 172.21.15.60:0/892842388 learned_addr learned my addr 172.21.15.60:0/892842388 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.049+0000 7f63967fc640 1 --2- 172.21.15.60:0/892842388 >> v2:172.21.15.114:3300/0 conn(0x7f63980b1040 0x7f63980b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.049+0000 7f63977fe640 1 -- 172.21.15.60:0/892842388 >> v2:172.21.15.114:3300/0 conn(0x7f63980b1040 msgr2=0x7f63980b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.049+0000 7f63977fe640 1 --2- 172.21.15.60:0/892842388 >> v2:172.21.15.114:3300/0 conn(0x7f63980b1040 0x7f63980b14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.049+0000 7f63977fe640 1 -- 172.21.15.60:0/892842388 >> v1:172.21.15.114:6789/0 conn(0x7f6398096d70 legacy=0x7f639809d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.049+0000 7f63977fe640 1 -- 172.21.15.60:0/892842388 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f639809c300 con 0x7f63980b2580 2024-05-09T05:01:56.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.050+0000 7f63977fe640 1 --2- 172.21.15.60:0/892842388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63980b2580 0x7f63980b29c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f63980743f0 tx=0x7f639002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7260db4ca039472e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:56.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.050+0000 7f6395ffb640 1 -- 172.21.15.60:0/892842388 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6390036070 con 0x7f63980b2580 2024-05-09T05:01:56.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.050+0000 7f6395ffb640 1 -- 172.21.15.60:0/892842388 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6390031600 con 0x7f63980b2580 2024-05-09T05:01:56.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.050+0000 7f6395ffb640 1 -- 172.21.15.60:0/892842388 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63900319d0 con 0x7f63980b2580 2024-05-09T05:01:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.051+0000 7f639d82a640 1 -- 172.21.15.60:0/892842388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63980b2580 msgr2=0x7f63980b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.051+0000 7f639d82a640 1 --2- 172.21.15.60:0/892842388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63980b2580 0x7f63980b29c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f63980743f0 tx=0x7f639002a6a0 comp rx=0 tx=0).stop 2024-05-09T05:01:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.051+0000 7f639d82a640 1 -- 172.21.15.60:0/892842388 shutdown_connections 2024-05-09T05:01:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.051+0000 7f639d82a640 1 --2- 172.21.15.60:0/892842388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63980b2580 0x7f63980b29c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.051+0000 7f639d82a640 1 --2- 172.21.15.60:0/892842388 >> v2:172.21.15.114:3300/0 conn(0x7f63980b1040 0x7f63980b14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.051+0000 7f639d82a640 1 -- 172.21.15.60:0/892842388 >> 172.21.15.60:0/892842388 conn(0x7f6398072390 msgr2=0x7f63980727a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.051+0000 7f639d82a640 1 -- 172.21.15.60:0/892842388 shutdown_connections 2024-05-09T05:01:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.051+0000 7f639d82a640 1 -- 172.21.15.60:0/892842388 wait complete. 2024-05-09T05:01:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.052+0000 7f639d82a640 1 Processor -- start 2024-05-09T05:01:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.052+0000 7f639d82a640 1 -- start start 2024-05-09T05:01:56.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.053+0000 7f639d82a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6398096d70 0x7f63981f5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:56.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.053+0000 7f639d82a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63980b1040 0x7f6398095a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:56.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.053+0000 7f639d82a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63980b2580 0x7f6398091980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:56.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.053+0000 7f639d82a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f639809cba0 con 0x7f63980b1040 2024-05-09T05:01:56.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.053+0000 7f639d82a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f639809c8c0 con 0x7f63980b2580 2024-05-09T05:01:56.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.053+0000 7f639d82a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f63980b19f0 con 0x7f6398096d70 2024-05-09T05:01:56.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.053+0000 7f63967fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63980b1040 0x7f6398095a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:56.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.053+0000 7f63967fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63980b1040 0x7f6398095a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35820/0 (socket says 172.21.15.60:35820) 2024-05-09T05:01:56.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.053+0000 7f63967fc640 1 -- 172.21.15.60:0/3598427453 learned_addr learned my addr 172.21.15.60:0/3598427453 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:56.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.053+0000 7f6396ffd640 1 --2- 172.21.15.60:0/3598427453 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6398096d70 0x7f63981f5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:56.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.053+0000 7f63977fe640 1 --2- 172.21.15.60:0/3598427453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63980b2580 0x7f6398091980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:56.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.053+0000 7f63967fc640 1 -- 172.21.15.60:0/3598427453 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6398096d70 msgr2=0x7f63981f5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:56.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.053+0000 7f63967fc640 1 --2- 172.21.15.60:0/3598427453 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6398096d70 0x7f63981f5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:56.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.053+0000 7f63967fc640 1 -- 172.21.15.60:0/3598427453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63980b2580 msgr2=0x7f6398091980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:56.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.053+0000 7f63967fc640 1 --2- 172.21.15.60:0/3598427453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63980b2580 0x7f6398091980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:56.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.053+0000 7f63967fc640 1 -- 172.21.15.60:0/3598427453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6398096370 con 0x7f63980b1040 2024-05-09T05:01:56.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.054+0000 7f63967fc640 1 --2- 172.21.15.60:0/3598427453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63980b1040 0x7f6398095a00 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f638c00c820 tx=0x7f638c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:56.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.054+0000 7f637ffff640 1 -- 172.21.15.60:0/3598427453 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c015070 con 0x7f63980b1040 2024-05-09T05:01:56.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.054+0000 7f637ffff640 1 -- 172.21.15.60:0/3598427453 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f638c004240 con 0x7f63980b1040 2024-05-09T05:01:56.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.054+0000 7f639d82a640 1 -- 172.21.15.60:0/3598427453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6398096090 con 0x7f63980b1040 2024-05-09T05:01:56.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.054+0000 7f637ffff640 1 -- 172.21.15.60:0/3598427453 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c00ce10 con 0x7f63980b1040 2024-05-09T05:01:56.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.054+0000 7f639d82a640 1 -- 172.21.15.60:0/3598427453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63981f5080 con 0x7f63980b1040 2024-05-09T05:01:56.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.055+0000 7f639d82a640 1 -- 172.21.15.60:0/3598427453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63980967d0 con 0x7f63980b1040 2024-05-09T05:01:56.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.056+0000 7f637ffff640 1 -- 172.21.15.60:0/3598427453 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f638c007830 con 0x7f63980b1040 2024-05-09T05:01:56.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.056+0000 7f637ffff640 1 --2- 172.21.15.60:0/3598427453 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f63840762d0 0x7f6384078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:56.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.056+0000 7f637ffff640 1 -- 172.21.15.60:0/3598427453 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6997+0+0 (secure 0 0 0) 0x7f638c097980 con 0x7f63980b1040 2024-05-09T05:01:56.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.056+0000 7f6396ffd640 1 --2- 172.21.15.60:0/3598427453 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f63840762d0 0x7f6384078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:56.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.057+0000 7f6396ffd640 1 --2- 172.21.15.60:0/3598427453 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f63840762d0 0x7f6384078790 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6398070380 tx=0x7f6388009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:56.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.060+0000 7f637ffff640 1 -- 172.21.15.60:0/3598427453 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f63980967d0 con 0x7f63980b1040 2024-05-09T05:01:56.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.289+0000 7f639d82a640 1 -- 172.21.15.60:0/3598427453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f6398051040 con 0x7f63980b1040 2024-05-09T05:01:56.955 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.954+0000 7f637ffff640 1 -- 172.21.15.60:0/3598427453 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v31) ==== 104+0+0 (secure 0 0 0) 0x7f638c060dd0 con 0x7f63980b1040 2024-05-09T05:01:56.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.957+0000 7f639d82a640 1 -- 172.21.15.60:0/3598427453 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f63840762d0 msgr2=0x7f6384078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:56.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.957+0000 7f639d82a640 1 --2- 172.21.15.60:0/3598427453 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f63840762d0 0x7f6384078790 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6398070380 tx=0x7f6388009040 comp rx=0 tx=0).stop 2024-05-09T05:01:56.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.957+0000 7f639d82a640 1 -- 172.21.15.60:0/3598427453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63980b1040 msgr2=0x7f6398095a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:56.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.957+0000 7f639d82a640 1 --2- 172.21.15.60:0/3598427453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63980b1040 0x7f6398095a00 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f638c00c820 tx=0x7f638c00cce0 comp rx=0 tx=0).stop 2024-05-09T05:01:56.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.957+0000 7f639d82a640 1 -- 172.21.15.60:0/3598427453 shutdown_connections 2024-05-09T05:01:56.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.957+0000 7f639d82a640 1 --2- 172.21.15.60:0/3598427453 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f63840762d0 0x7f6384078790 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:56.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.957+0000 7f639d82a640 1 --2- 172.21.15.60:0/3598427453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63980b2580 0x7f6398091980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:56.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.957+0000 7f639d82a640 1 --2- 172.21.15.60:0/3598427453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63980b1040 0x7f6398095a00 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:56.958 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.957+0000 7f639d82a640 1 --2- 172.21.15.60:0/3598427453 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6398096d70 0x7f63981f5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:56.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.957+0000 7f639d82a640 1 -- 172.21.15.60:0/3598427453 >> 172.21.15.60:0/3598427453 conn(0x7f6398072390 msgr2=0x7f6398097360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:56.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.957+0000 7f639d82a640 1 -- 172.21.15.60:0/3598427453 shutdown_connections 2024-05-09T05:01:56.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:56.957+0000 7f639d82a640 1 -- 172.21.15.60:0/3598427453 wait complete. 2024-05-09T05:01:56.971 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-09T05:01:57.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.123+0000 7f86659dd640 1 Processor -- start 2024-05-09T05:01:57.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.123+0000 7f86659dd640 1 -- start start 2024-05-09T05:01:57.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.123+0000 7f86659dd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8660145160 0x7f86600cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:57.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.123+0000 7f86659dd640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86600ce610 con 0x7f86600d0de0 2024-05-09T05:01:57.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.123+0000 7f86659dd640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8660071810 con 0x7f8660070860 2024-05-09T05:01:57.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.123+0000 7f86659dd640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8660071af0 con 0x7f8660145160 2024-05-09T05:01:57.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865e7fc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f86600d0de0 0x7f8660070150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57194/0 (socket says 172.21.15.60:57194) 2024-05-09T05:01:57.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865e7fc640 1 -- 172.21.15.60:0/870829221 learned_addr learned my addr 172.21.15.60:0/870829221 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:57.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865effd640 1 --2- 172.21.15.60:0/870829221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8660145160 0x7f86600cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:57.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865dffb640 1 -- 172.21.15.60:0/870829221 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3875037182 0 0) 0x7f86600ce610 con 0x7f86600d0de0 2024-05-09T05:01:57.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865dffb640 1 -- 172.21.15.60:0/870829221 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f864c003520 con 0x7f86600d0de0 2024-05-09T05:01:57.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865dffb640 1 -- 172.21.15.60:0/870829221 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2382044545 0 0) 0x7f864c003520 con 0x7f86600d0de0 2024-05-09T05:01:57.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865dffb640 1 -- 172.21.15.60:0/870829221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8660145160 msgr2=0x7f86600cdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:57.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865dffb640 1 --2- 172.21.15.60:0/870829221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8660145160 0x7f86600cdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:57.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865dffb640 1 -- 172.21.15.60:0/870829221 >> v1:172.21.15.114:6789/0 conn(0x7f8660070860 legacy=0x7f86600725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:57.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865dffb640 1 -- 172.21.15.60:0/870829221 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8660071dd0 con 0x7f86600d0de0 2024-05-09T05:01:57.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865dffb640 1 -- 172.21.15.60:0/870829221 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f86480016f0 con 0x7f86600d0de0 2024-05-09T05:01:57.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865dffb640 1 -- 172.21.15.60:0/870829221 >> v1:172.21.15.60:6789/0 conn(0x7f86600d0de0 legacy=0x7f8660070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:57.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865dffb640 1 --2- 172.21.15.60:0/870829221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f864c003a80 0x7f864c003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:57.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865dffb640 1 --2- 172.21.15.60:0/870829221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f864c004800 0x7f864c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:57.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865dffb640 1 --2- 172.21.15.60:0/870829221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8660145160 0x7f864c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:57.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865dffb640 1 -- 172.21.15.60:0/870829221 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8660071af0 con 0x7f864c003a80 2024-05-09T05:01:57.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865dffb640 1 -- 172.21.15.60:0/870829221 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f864c005520 con 0x7f864c004800 2024-05-09T05:01:57.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865dffb640 1 -- 172.21.15.60:0/870829221 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f864c00e110 con 0x7f8660145160 2024-05-09T05:01:57.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865effd640 1 --2- 172.21.15.60:0/870829221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f864c003a80 0x7f864c003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:57.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865e7fc640 1 --2- 172.21.15.60:0/870829221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f864c004800 0x7f864c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:57.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.124+0000 7f865f7fe640 1 --2- 172.21.15.60:0/870829221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8660145160 0x7f864c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:57.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.125+0000 7f865effd640 1 -- 172.21.15.60:0/870829221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8660145160 msgr2=0x7f864c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:57.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.125+0000 7f865effd640 1 --2- 172.21.15.60:0/870829221 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8660145160 0x7f864c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:57.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.125+0000 7f865effd640 1 -- 172.21.15.60:0/870829221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f864c004800 msgr2=0x7f864c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:57.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.125+0000 7f865effd640 1 --2- 172.21.15.60:0/870829221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f864c004800 0x7f864c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:57.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.125+0000 7f865effd640 1 -- 172.21.15.60:0/870829221 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864c003520 con 0x7f864c003a80 2024-05-09T05:01:57.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.125+0000 7f865effd640 1 --2- 172.21.15.60:0/870829221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f864c003a80 0x7f864c003ea0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f8650009fc0 tx=0x7f864c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33fc3a9f98e9493a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:57.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.126+0000 7f865dffb640 1 -- 172.21.15.60:0/870829221 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8650009c10 con 0x7f864c003a80 2024-05-09T05:01:57.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.126+0000 7f865dffb640 1 -- 172.21.15.60:0/870829221 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8650036070 con 0x7f864c003a80 2024-05-09T05:01:57.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.126+0000 7f865dffb640 1 -- 172.21.15.60:0/870829221 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8650031880 con 0x7f864c003a80 2024-05-09T05:01:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.126+0000 7f86659dd640 1 -- 172.21.15.60:0/870829221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f864c003a80 msgr2=0x7f864c003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.126+0000 7f86659dd640 1 --2- 172.21.15.60:0/870829221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f864c003a80 0x7f864c003ea0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f8650009fc0 tx=0x7f864c005830 comp rx=0 tx=0).stop 2024-05-09T05:01:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.126+0000 7f865f7fe640 1 -- 172.21.15.60:0/870829221 reap_dead start 2024-05-09T05:01:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.126+0000 7f86659dd640 1 -- 172.21.15.60:0/870829221 shutdown_connections 2024-05-09T05:01:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.126+0000 7f86659dd640 1 -- 172.21.15.60:0/870829221 >> 172.21.15.60:0/870829221 conn(0x7f86600b0f60 msgr2=0x7f86600b1370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.126+0000 7f86659dd640 1 -- 172.21.15.60:0/870829221 shutdown_connections 2024-05-09T05:01:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.126+0000 7f86659dd640 1 -- 172.21.15.60:0/870829221 wait complete. 2024-05-09T05:01:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.127+0000 7f86659dd640 1 Processor -- start 2024-05-09T05:01:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.128+0000 7f86659dd640 1 -- start start 2024-05-09T05:01:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.128+0000 7f86659dd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f864c003a80 0x7f866015e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.128+0000 7f86659dd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f866015e5a0 0x7f866015ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:57.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.128+0000 7f86659dd640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f866015ef60 0x7f86600ae890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.128+0000 7f86659dd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f86600cdd10 con 0x7f866015e5a0 2024-05-09T05:01:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.128+0000 7f86659dd640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8660145540 con 0x7f864c003a80 2024-05-09T05:01:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.128+0000 7f86659dd640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f86600cde90 con 0x7f866015ef60 2024-05-09T05:01:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.128+0000 7f865e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f866015e5a0 0x7f866015ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.128+0000 7f865e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f866015e5a0 0x7f866015ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35834/0 (socket says 172.21.15.60:35834) 2024-05-09T05:01:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.128+0000 7f865e7fc640 1 -- 172.21.15.60:0/3155773978 learned_addr learned my addr 172.21.15.60:0/3155773978 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.129+0000 7f865f7fe640 1 --2- 172.21.15.60:0/3155773978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f866015ef60 0x7f86600ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.129+0000 7f865effd640 1 --2- 172.21.15.60:0/3155773978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f864c003a80 0x7f866015e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.129+0000 7f865e7fc640 1 -- 172.21.15.60:0/3155773978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f866015ef60 msgr2=0x7f86600ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.129+0000 7f865e7fc640 1 --2- 172.21.15.60:0/3155773978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f866015ef60 0x7f86600ae890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.129+0000 7f865e7fc640 1 -- 172.21.15.60:0/3155773978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f864c003a80 msgr2=0x7f866015e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.129+0000 7f865e7fc640 1 --2- 172.21.15.60:0/3155773978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f864c003a80 0x7f866015e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.129+0000 7f865e7fc640 1 -- 172.21.15.60:0/3155773978 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f866015cfc0 con 0x7f866015e5a0 2024-05-09T05:01:57.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.129+0000 7f865f7fe640 1 --2- 172.21.15.60:0/3155773978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f866015ef60 0x7f86600ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:01:57.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.129+0000 7f865e7fc640 1 --2- 172.21.15.60:0/3155773978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f866015e5a0 0x7f866015ea20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8648007340 tx=0x7f8648007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:57.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.129+0000 7f8657fff640 1 -- 172.21.15.60:0/3155773978 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864800f940 con 0x7f866015e5a0 2024-05-09T05:01:57.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.129+0000 7f8657fff640 1 -- 172.21.15.60:0/3155773978 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8648018070 con 0x7f866015e5a0 2024-05-09T05:01:57.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.129+0000 7f86659dd640 1 -- 172.21.15.60:0/3155773978 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f866015d2a0 con 0x7f866015e5a0 2024-05-09T05:01:57.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.129+0000 7f8657fff640 1 -- 172.21.15.60:0/3155773978 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864800fd40 con 0x7f866015e5a0 2024-05-09T05:01:57.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.129+0000 7f86659dd640 1 -- 172.21.15.60:0/3155773978 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f866015d860 con 0x7f866015e5a0 2024-05-09T05:01:57.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.130+0000 7f86659dd640 1 -- 172.21.15.60:0/3155773978 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8660051040 con 0x7f866015e5a0 2024-05-09T05:01:57.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.131+0000 7f8657fff640 1 -- 172.21.15.60:0/3155773978 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f864801d460 con 0x7f866015e5a0 2024-05-09T05:01:57.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.131+0000 7f8657fff640 1 --2- 172.21.15.60:0/3155773978 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f86400762d0 0x7f8640078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:57.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.131+0000 7f8657fff640 1 -- 172.21.15.60:0/3155773978 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6997+0+0 (secure 0 0 0) 0x7f8648097ab0 con 0x7f866015e5a0 2024-05-09T05:01:57.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.131+0000 7f865effd640 1 --2- 172.21.15.60:0/3155773978 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f86400762d0 0x7f8640078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:57.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.132+0000 7f865effd640 1 --2- 172.21.15.60:0/3155773978 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f86400762d0 0x7f8640078790 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8650035000 tx=0x7f865002f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:57.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.136+0000 7f8657fff640 1 -- 172.21.15.60:0/3155773978 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8648061050 con 0x7f866015e5a0 2024-05-09T05:01:57.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.327+0000 7f86659dd640 1 -- 172.21.15.60:0/3155773978 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f86600b0d40 con 0x7f866015e5a0 2024-05-09T05:01:57.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.959+0000 7f8657fff640 1 -- 172.21.15.60:0/3155773978 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v59) ==== 181+0+0 (secure 0 0 0) 0x7f8648060950 con 0x7f866015e5a0 2024-05-09T05:01:57.960 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2024-05-09T05:01:57.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.963+0000 7f86659dd640 1 -- 172.21.15.60:0/3155773978 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f86400762d0 msgr2=0x7f8640078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:57.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.963+0000 7f86659dd640 1 --2- 172.21.15.60:0/3155773978 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f86400762d0 0x7f8640078790 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8650035000 tx=0x7f865002f000 comp rx=0 tx=0).stop 2024-05-09T05:01:57.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.963+0000 7f86659dd640 1 -- 172.21.15.60:0/3155773978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f866015e5a0 msgr2=0x7f866015ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:57.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.963+0000 7f86659dd640 1 --2- 172.21.15.60:0/3155773978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f866015e5a0 0x7f866015ea20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8648007340 tx=0x7f8648007800 comp rx=0 tx=0).stop 2024-05-09T05:01:57.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.963+0000 7f86659dd640 1 -- 172.21.15.60:0/3155773978 shutdown_connections 2024-05-09T05:01:57.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.963+0000 7f86659dd640 1 --2- 172.21.15.60:0/3155773978 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f86400762d0 0x7f8640078790 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:57.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.963+0000 7f86659dd640 1 --2- 172.21.15.60:0/3155773978 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f866015ef60 0x7f86600ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:57.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.964+0000 7f86659dd640 1 --2- 172.21.15.60:0/3155773978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f866015e5a0 0x7f866015ea20 secure :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8648007340 tx=0x7f8648007800 comp rx=0 tx=0).stop 2024-05-09T05:01:57.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.964+0000 7f86659dd640 1 --2- 172.21.15.60:0/3155773978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f864c003a80 0x7f866015e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:57.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.964+0000 7f86659dd640 1 -- 172.21.15.60:0/3155773978 >> 172.21.15.60:0/3155773978 conn(0x7f86600b0f60 msgr2=0x7f86600b21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:57.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.965+0000 7f86659dd640 1 -- 172.21.15.60:0/3155773978 shutdown_connections 2024-05-09T05:01:57.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:57.965+0000 7f86659dd640 1 -- 172.21.15.60:0/3155773978 wait complete. 2024-05-09T05:01:57.979 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-09T05:01:58.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.100+0000 7ffbcca3e640 1 Processor -- start 2024-05-09T05:01:58.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.100+0000 7ffbcca3e640 1 -- start start 2024-05-09T05:01:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.101+0000 7ffbcca3e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffbc8160600 0x7ffbc815d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:58.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.101+0000 7ffbcca3e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbc80d0f70 con 0x7ffbc80b4770 2024-05-09T05:01:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.101+0000 7ffbcca3e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbc80d1230 con 0x7ffbc80b8f40 2024-05-09T05:01:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.101+0000 7ffbcca3e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7ffbc80d1510 con 0x7ffbc8160600 2024-05-09T05:01:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.101+0000 7ffbc659c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffbc8160600 0x7ffbc815d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.101+0000 7ffbc6d9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ffbc80b4770 0x7ffbc815eae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57198/0 (socket says 172.21.15.60:57198) 2024-05-09T05:01:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.101+0000 7ffbc6d9d640 1 -- 172.21.15.60:0/1799120068 learned_addr learned my addr 172.21.15.60:0/1799120068 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.101+0000 7ffbc659c640 1 -- 172.21.15.60:0/1799120068 >> v1:172.21.15.114:6789/0 conn(0x7ffbc80b8f40 legacy=0x7ffbc80b4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.101+0000 7ffbc659c640 1 -- 172.21.15.60:0/1799120068 >> v1:172.21.15.60:6789/0 conn(0x7ffbc80b4770 legacy=0x7ffbc815eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.101+0000 7ffbc659c640 1 -- 172.21.15.60:0/1799120068 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbc80d17f0 con 0x7ffbc8160600 2024-05-09T05:01:58.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.101+0000 7ffbc559a640 1 -- 172.21.15.60:0/1799120068 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1112106961 0 0) 0x7ffbc80d0f70 con 0x7ffbc80b4770 2024-05-09T05:01:58.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.102+0000 7ffbc659c640 1 --2- 172.21.15.60:0/1799120068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbc8160600 0x7ffbc815d6e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80097d0 tx=0x7ffbb802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902df18aafe410f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.102+0000 7ffbc559a640 1 -- 172.21.15.60:0/1799120068 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb8037070 con 0x7ffbc8160600 2024-05-09T05:01:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.102+0000 7ffbc559a640 1 -- 172.21.15.60:0/1799120068 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbb8032440 con 0x7ffbc8160600 2024-05-09T05:01:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.102+0000 7ffbc559a640 1 -- 172.21.15.60:0/1799120068 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb8032810 con 0x7ffbc8160600 2024-05-09T05:01:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.103+0000 7ffbcca3e640 1 -- 172.21.15.60:0/1799120068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbc8160600 msgr2=0x7ffbc815d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.103+0000 7ffbcca3e640 1 --2- 172.21.15.60:0/1799120068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbc8160600 0x7ffbc815d6e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80097d0 tx=0x7ffbb802a270 comp rx=0 tx=0).stop 2024-05-09T05:01:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.103+0000 7ffbcca3e640 1 -- 172.21.15.60:0/1799120068 shutdown_connections 2024-05-09T05:01:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.103+0000 7ffbcca3e640 1 --2- 172.21.15.60:0/1799120068 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbc8160600 0x7ffbc815d6e0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.103+0000 7ffbcca3e640 1 -- 172.21.15.60:0/1799120068 >> 172.21.15.60:0/1799120068 conn(0x7ffbc80b3c80 msgr2=0x7ffbc8070590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.103+0000 7ffbcca3e640 1 -- 172.21.15.60:0/1799120068 shutdown_connections 2024-05-09T05:01:58.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.103+0000 7ffbcca3e640 1 -- 172.21.15.60:0/1799120068 wait complete. 2024-05-09T05:01:58.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.104+0000 7ffbcca3e640 1 Processor -- start 2024-05-09T05:01:58.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.104+0000 7ffbcca3e640 1 -- start start 2024-05-09T05:01:58.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.105+0000 7ffbcca3e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc80b4770 0x7ffbc81570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:58.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.105+0000 7ffbcca3e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbc80b8f40 0x7ffbc81f2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:58.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.105+0000 7ffbcca3e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc8160600 0x7ffbc81528d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:58.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.105+0000 7ffbcca3e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc81609e0 con 0x7ffbc80b8f40 2024-05-09T05:01:58.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.105+0000 7ffbcca3e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc80d0c60 con 0x7ffbc80b4770 2024-05-09T05:01:58.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.105+0000 7ffbcca3e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ffbc815f1f0 con 0x7ffbc8160600 2024-05-09T05:01:58.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.105+0000 7ffbc5d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbc80b8f40 0x7ffbc81f2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:58.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.105+0000 7ffbc5d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbc80b8f40 0x7ffbc81f2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35848/0 (socket says 172.21.15.60:35848) 2024-05-09T05:01:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.105+0000 7ffbc5d9b640 1 -- 172.21.15.60:0/2919402737 learned_addr learned my addr 172.21.15.60:0/2919402737 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.105+0000 7ffbc6d9d640 1 --2- 172.21.15.60:0/2919402737 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc8160600 0x7ffbc81528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.105+0000 7ffbc659c640 1 --2- 172.21.15.60:0/2919402737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc80b4770 0x7ffbc81570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.105+0000 7ffbc5d9b640 1 -- 172.21.15.60:0/2919402737 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc8160600 msgr2=0x7ffbc81528d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.105+0000 7ffbc5d9b640 1 --2- 172.21.15.60:0/2919402737 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc8160600 0x7ffbc81528d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.105+0000 7ffbc5d9b640 1 -- 172.21.15.60:0/2919402737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc80b4770 msgr2=0x7ffbc81570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.105+0000 7ffbc5d9b640 1 --2- 172.21.15.60:0/2919402737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc80b4770 0x7ffbc81570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.105+0000 7ffbc5d9b640 1 -- 172.21.15.60:0/2919402737 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbc81f35e0 con 0x7ffbc80b8f40 2024-05-09T05:01:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.106+0000 7ffbc5d9b640 1 --2- 172.21.15.60:0/2919402737 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbc80b8f40 0x7ffbc81f2d90 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc00c8a0 tx=0x7ffbbc00cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.106+0000 7ffbb6ffd640 1 -- 172.21.15.60:0/2919402737 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc003b70 con 0x7ffbc80b8f40 2024-05-09T05:01:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.106+0000 7ffbb6ffd640 1 -- 172.21.15.60:0/2919402737 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbbc003d10 con 0x7ffbc80b8f40 2024-05-09T05:01:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.106+0000 7ffbcca3e640 1 -- 172.21.15.60:0/2919402737 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbc81f38c0 con 0x7ffbc80b8f40 2024-05-09T05:01:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.106+0000 7ffbb6ffd640 1 -- 172.21.15.60:0/2919402737 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc00aa70 con 0x7ffbc80b8f40 2024-05-09T05:01:58.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.106+0000 7ffbcca3e640 1 -- 172.21.15.60:0/2919402737 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbc81f4180 con 0x7ffbc80b8f40 2024-05-09T05:01:58.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.107+0000 7ffbcca3e640 1 -- 172.21.15.60:0/2919402737 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbc81f4380 con 0x7ffbc80b8f40 2024-05-09T05:01:58.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.108+0000 7ffbb6ffd640 1 -- 172.21.15.60:0/2919402737 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ffbbc00ac10 con 0x7ffbc80b8f40 2024-05-09T05:01:58.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.108+0000 7ffbb6ffd640 1 --2- 172.21.15.60:0/2919402737 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ffba80762d0 0x7ffba8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:58.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.108+0000 7ffbb6ffd640 1 -- 172.21.15.60:0/2919402737 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6559+0+0 (secure 0 0 0) 0x7ffbbc0977c0 con 0x7ffbc80b8f40 2024-05-09T05:01:58.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.108+0000 7ffbc659c640 1 --2- 172.21.15.60:0/2919402737 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ffba80762d0 0x7ffba8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:58.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.109+0000 7ffbc659c640 1 --2- 172.21.15.60:0/2919402737 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ffba80762d0 0x7ffba8078790 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8034000 tx=0x7ffbb802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:58.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.113+0000 7ffbb6ffd640 1 -- 172.21.15.60:0/2919402737 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ffbc81f4380 con 0x7ffbc80b8f40 2024-05-09T05:01:58.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.304+0000 7ffbcca3e640 1 -- 172.21.15.60:0/2919402737 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7ffbc80d22c0 con 0x7ffbc80b8f40 2024-05-09T05:01:58.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.966+0000 7ffbb6ffd640 1 -- 172.21.15.60:0/2919402737 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v60) ==== 169+0+0 (secure 0 0 0) 0x7ffbbc060780 con 0x7ffbc80b8f40 2024-05-09T05:01:58.967 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2024-05-09T05:01:58.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.970+0000 7ffbcca3e640 1 -- 172.21.15.60:0/2919402737 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ffba80762d0 msgr2=0x7ffba8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:58.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.970+0000 7ffbcca3e640 1 --2- 172.21.15.60:0/2919402737 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ffba80762d0 0x7ffba8078790 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8034000 tx=0x7ffbb802e000 comp rx=0 tx=0).stop 2024-05-09T05:01:58.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.970+0000 7ffbcca3e640 1 -- 172.21.15.60:0/2919402737 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbc80b8f40 msgr2=0x7ffbc81f2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:58.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.970+0000 7ffbcca3e640 1 --2- 172.21.15.60:0/2919402737 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbc80b8f40 0x7ffbc81f2d90 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc00c8a0 tx=0x7ffbbc00cd60 comp rx=0 tx=0).stop 2024-05-09T05:01:58.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.971+0000 7ffbcca3e640 1 -- 172.21.15.60:0/2919402737 shutdown_connections 2024-05-09T05:01:58.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.971+0000 7ffbcca3e640 1 --2- 172.21.15.60:0/2919402737 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ffba80762d0 0x7ffba8078790 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:58.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.971+0000 7ffbcca3e640 1 --2- 172.21.15.60:0/2919402737 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ffbc8160600 0x7ffbc81528d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:58.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.971+0000 7ffbcca3e640 1 --2- 172.21.15.60:0/2919402737 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffbc80b8f40 0x7ffbc81f2d90 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:58.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.971+0000 7ffbcca3e640 1 --2- 172.21.15.60:0/2919402737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffbc80b4770 0x7ffbc81570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:58.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.971+0000 7ffbcca3e640 1 -- 172.21.15.60:0/2919402737 >> 172.21.15.60:0/2919402737 conn(0x7ffbc80b3c80 msgr2=0x7ffbc81613f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:58.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.972+0000 7ffbcca3e640 1 -- 172.21.15.60:0/2919402737 shutdown_connections 2024-05-09T05:01:58.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:58.972+0000 7ffbcca3e640 1 -- 172.21.15.60:0/2919402737 wait complete. 2024-05-09T05:01:58.987 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_extra1 cephfs_data_extra1 --yes-i-really-really-mean-it 2024-05-09T05:01:59.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.102+0000 7f96bcd04640 1 Processor -- start 2024-05-09T05:01:59.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.102+0000 7f96bcd04640 1 -- start start 2024-05-09T05:01:59.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.103+0000 7f96bcd04640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b81602a0 0x7f96b80b4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:59.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.103+0000 7f96bcd04640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f96b80d2530 0x7f96b80cffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:59.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.103+0000 7f96bcd04640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96b80d2b80 con 0x7f96b80b4860 2024-05-09T05:01:59.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.103+0000 7f96bcd04640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f96b80b2d50 con 0x7f96b80d2530 2024-05-09T05:01:59.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.103+0000 7f96bcd04640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f96b806c510 con 0x7f96b81602a0 2024-05-09T05:01:59.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.103+0000 7f96affff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f96b80b4860 0x7f96b80d1e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34236/0 (socket says 172.21.15.60:34236) 2024-05-09T05:01:59.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.103+0000 7f96affff640 1 -- 172.21.15.60:0/1978741513 learned_addr learned my addr 172.21.15.60:0/1978741513 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:01:59.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.103+0000 7f96b5d9b640 1 -- 172.21.15.60:0/1978741513 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3548342957 0 0) 0x7f96b80d2b80 con 0x7f96b80b4860 2024-05-09T05:01:59.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.103+0000 7f96b659c640 1 --2- 172.21.15.60:0/1978741513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b81602a0 0x7f96b80b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:59.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.103+0000 7f96b6d9d640 1 --2- 172.21.15.60:0/1978741513 >> v2:172.21.15.114:3300/0 conn(0x7f96b80d2530 0x7f96b80cffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:59.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.103+0000 7f96b5d9b640 1 -- 172.21.15.60:0/1978741513 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96a4003520 con 0x7f96b80b4860 2024-05-09T05:01:59.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b5d9b640 1 -- 172.21.15.60:0/1978741513 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1036355235 0 0) 0x7f96a4003520 con 0x7f96b80b4860 2024-05-09T05:01:59.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b5d9b640 1 -- 172.21.15.60:0/1978741513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b81602a0 msgr2=0x7f96b80b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:59.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b5d9b640 1 --2- 172.21.15.60:0/1978741513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b81602a0 0x7f96b80b4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:59.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b5d9b640 1 -- 172.21.15.60:0/1978741513 >> v2:172.21.15.114:3300/0 conn(0x7f96b80d2530 msgr2=0x7f96b80cffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:59.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b5d9b640 1 --2- 172.21.15.60:0/1978741513 >> v2:172.21.15.114:3300/0 conn(0x7f96b80d2530 0x7f96b80cffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:59.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b5d9b640 1 -- 172.21.15.60:0/1978741513 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b806c7f0 con 0x7f96b80b4860 2024-05-09T05:01:59.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b5d9b640 1 -- 172.21.15.60:0/1978741513 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f96a00016f0 con 0x7f96b80b4860 2024-05-09T05:01:59.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b5d9b640 1 -- 172.21.15.60:0/1978741513 >> v1:172.21.15.60:6789/0 conn(0x7f96b80b4860 legacy=0x7f96b80d1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:59.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b5d9b640 1 --2- 172.21.15.60:0/1978741513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b81602a0 0x7f96a4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:59.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b5d9b640 1 --2- 172.21.15.60:0/1978741513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96a4003fd0 0x7f96a4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:59.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b5d9b640 1 --2- 172.21.15.60:0/1978741513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96a4004d40 0x7f96a40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:59.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b5d9b640 1 -- 172.21.15.60:0/1978741513 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f96b80b2d50 con 0x7f96a4004d40 2024-05-09T05:01:59.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b5d9b640 1 -- 172.21.15.60:0/1978741513 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f96b806c510 con 0x7f96a4003fd0 2024-05-09T05:01:59.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b5d9b640 1 -- 172.21.15.60:0/1978741513 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f96a400ba10 con 0x7f96b81602a0 2024-05-09T05:01:59.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b6d9d640 1 --2- 172.21.15.60:0/1978741513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96a4004d40 0x7f96a40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:59.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b659c640 1 --2- 172.21.15.60:0/1978741513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b81602a0 0x7f96a4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:59.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96affff640 1 --2- 172.21.15.60:0/1978741513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96a4003fd0 0x7f96a4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:59.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b6d9d640 1 -- 172.21.15.60:0/1978741513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b81602a0 msgr2=0x7f96a4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:59.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b6d9d640 1 --2- 172.21.15.60:0/1978741513 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b81602a0 0x7f96a4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:59.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b6d9d640 1 -- 172.21.15.60:0/1978741513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96a4003fd0 msgr2=0x7f96a4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:01:59.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b6d9d640 1 --2- 172.21.15.60:0/1978741513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96a4003fd0 0x7f96a4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:59.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.104+0000 7f96b6d9d640 1 -- 172.21.15.60:0/1978741513 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96a4003520 con 0x7f96a4004d40 2024-05-09T05:01:59.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.105+0000 7f96b6d9d640 1 --2- 172.21.15.60:0/1978741513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96a4004d40 0x7f96a40051c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f96b000e170 tx=0x7f96a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c69013439f84782d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:59.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.105+0000 7f96b5d9b640 1 -- 172.21.15.60:0/1978741513 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b0034020 con 0x7f96a4004d40 2024-05-09T05:01:59.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.105+0000 7f96b5d9b640 1 -- 172.21.15.60:0/1978741513 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96b0030430 con 0x7f96a4004d40 2024-05-09T05:01:59.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.105+0000 7f96b5d9b640 1 -- 172.21.15.60:0/1978741513 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b0009c10 con 0x7f96a4004d40 2024-05-09T05:01:59.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.106+0000 7f96bcd04640 1 -- 172.21.15.60:0/1978741513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96a4004d40 msgr2=0x7f96a40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:59.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.106+0000 7f96bcd04640 1 --2- 172.21.15.60:0/1978741513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96a4004d40 0x7f96a40051c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f96b000e170 tx=0x7f96a4005830 comp rx=0 tx=0).stop 2024-05-09T05:01:59.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.106+0000 7f96b6d9d640 1 -- 172.21.15.60:0/1978741513 reap_dead start 2024-05-09T05:01:59.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.106+0000 7f96bcd04640 1 -- 172.21.15.60:0/1978741513 shutdown_connections 2024-05-09T05:01:59.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.106+0000 7f96bcd04640 1 -- 172.21.15.60:0/1978741513 >> 172.21.15.60:0/1978741513 conn(0x7f96b80b2f00 msgr2=0x7f96b8070590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:59.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.106+0000 7f96bcd04640 1 -- 172.21.15.60:0/1978741513 shutdown_connections 2024-05-09T05:01:59.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.106+0000 7f96bcd04640 1 -- 172.21.15.60:0/1978741513 wait complete. 2024-05-09T05:01:59.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.107+0000 7f96bcd04640 1 Processor -- start 2024-05-09T05:01:59.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.107+0000 7f96bcd04640 1 -- start start 2024-05-09T05:01:59.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.108+0000 7f96bcd04640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96a4004d40 0x7f96b81f4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:59.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.108+0000 7f96bcd04640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b8154260 0x7f96b8150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:59.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.108+0000 7f96bcd04640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b8161840 0x7f96b8161cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:59.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.108+0000 7f96bcd04640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f96b806d2d0 con 0x7f96b8154260 2024-05-09T05:01:59.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.108+0000 7f96bcd04640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f96b806cdb0 con 0x7f96a4004d40 2024-05-09T05:01:59.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.108+0000 7f96bcd04640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f96b8160680 con 0x7f96b8161840 2024-05-09T05:01:59.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.108+0000 7f96affff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b8154260 0x7f96b8150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:59.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.108+0000 7f96affff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b8154260 0x7f96b8150cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36222/0 (socket says 172.21.15.60:36222) 2024-05-09T05:01:59.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.108+0000 7f96affff640 1 -- 172.21.15.60:0/390445338 learned_addr learned my addr 172.21.15.60:0/390445338 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:01:59.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.108+0000 7f96b6d9d640 1 --2- 172.21.15.60:0/390445338 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b8161840 0x7f96b8161cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:59.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.108+0000 7f96b659c640 1 --2- 172.21.15.60:0/390445338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96a4004d40 0x7f96b81f4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:59.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.108+0000 7f96affff640 1 -- 172.21.15.60:0/390445338 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b8161840 msgr2=0x7f96b8161cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:59.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.108+0000 7f96affff640 1 --2- 172.21.15.60:0/390445338 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b8161840 0x7f96b8161cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:59.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.108+0000 7f96affff640 1 -- 172.21.15.60:0/390445338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96a4004d40 msgr2=0x7f96b81f4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:59.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.108+0000 7f96affff640 1 --2- 172.21.15.60:0/390445338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96a4004d40 0x7f96b81f4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:59.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.108+0000 7f96affff640 1 -- 172.21.15.60:0/390445338 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b81f5540 con 0x7f96b8154260 2024-05-09T05:01:59.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.109+0000 7f96b6d9d640 1 --2- 172.21.15.60:0/390445338 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b8161840 0x7f96b8161cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:59.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.109+0000 7f96b659c640 1 --2- 172.21.15.60:0/390445338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96a4004d40 0x7f96b81f4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:01:59.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.109+0000 7f96affff640 1 --2- 172.21.15.60:0/390445338 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b8154260 0x7f96b8150cb0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f96a00072c0 tx=0x7f96a0007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:59.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.109+0000 7f96aeffd640 1 -- 172.21.15.60:0/390445338 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a000f9c0 con 0x7f96b8154260 2024-05-09T05:01:59.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.109+0000 7f96aeffd640 1 -- 172.21.15.60:0/390445338 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96a0018070 con 0x7f96b8154260 2024-05-09T05:01:59.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.109+0000 7f96bcd04640 1 -- 172.21.15.60:0/390445338 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b81f5820 con 0x7f96b8154260 2024-05-09T05:01:59.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.109+0000 7f96aeffd640 1 -- 172.21.15.60:0/390445338 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a000fda0 con 0x7f96b8154260 2024-05-09T05:01:59.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.109+0000 7f96bcd04640 1 -- 172.21.15.60:0/390445338 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b80ce150 con 0x7f96b8154260 2024-05-09T05:01:59.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.110+0000 7f96bcd04640 1 -- 172.21.15.60:0/390445338 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96b80ce350 con 0x7f96b8154260 2024-05-09T05:01:59.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.110+0000 7f96aeffd640 1 -- 172.21.15.60:0/390445338 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f96a001d460 con 0x7f96b8154260 2024-05-09T05:01:59.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.111+0000 7f96aeffd640 1 --2- 172.21.15.60:0/390445338 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9698076300 0x7f96980787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:01:59.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.111+0000 7f96aeffd640 1 -- 172.21.15.60:0/390445338 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6145+0+0 (secure 0 0 0) 0x7f96a0097940 con 0x7f96b8154260 2024-05-09T05:01:59.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.111+0000 7f96b659c640 1 --2- 172.21.15.60:0/390445338 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9698076300 0x7f96980787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:01:59.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.112+0000 7f96b659c640 1 --2- 172.21.15.60:0/390445338 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9698076300 0x7f96980787c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009ab0 tx=0x7f96a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:01:59.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.115+0000 7f96aeffd640 1 -- 172.21.15.60:0/390445338 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f96b80ce350 con 0x7f96b8154260 2024-05-09T05:01:59.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.306+0000 7f96bcd04640 1 -- 172.21.15.60:0/390445338 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_extra1", "pool2": "cephfs_data_extra1", "yes_i_really_really_mean_it": true} v 0) -- 0x7f96b8051040 con 0x7f96b8154260 2024-05-09T05:01:59.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.968+0000 7f96aeffd640 1 -- 172.21.15.60:0/390445338 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_extra1", "pool2": "cephfs_data_extra1", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_extra1' removed v61) ==== 190+0+0 (secure 0 0 0) 0x7f96a00610e0 con 0x7f96b8154260 2024-05-09T05:01:59.968 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data_extra1' removed 2024-05-09T05:01:59.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.974+0000 7f96bcd04640 1 -- 172.21.15.60:0/390445338 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9698076300 msgr2=0x7f96980787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:59.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.974+0000 7f96bcd04640 1 --2- 172.21.15.60:0/390445338 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9698076300 0x7f96980787c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009ab0 tx=0x7f96a8008040 comp rx=0 tx=0).stop 2024-05-09T05:01:59.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.974+0000 7f96bcd04640 1 -- 172.21.15.60:0/390445338 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b8154260 msgr2=0x7f96b8150cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:01:59.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.974+0000 7f96bcd04640 1 --2- 172.21.15.60:0/390445338 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b8154260 0x7f96b8150cb0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f96a00072c0 tx=0x7f96a0007780 comp rx=0 tx=0).stop 2024-05-09T05:01:59.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.975+0000 7f96bcd04640 1 -- 172.21.15.60:0/390445338 shutdown_connections 2024-05-09T05:01:59.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.975+0000 7f96bcd04640 1 --2- 172.21.15.60:0/390445338 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9698076300 0x7f96980787c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:59.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.975+0000 7f96bcd04640 1 --2- 172.21.15.60:0/390445338 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f96b8161840 0x7f96b8161cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:59.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.975+0000 7f96bcd04640 1 --2- 172.21.15.60:0/390445338 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96b8154260 0x7f96b8150cb0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:59.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.975+0000 7f96bcd04640 1 --2- 172.21.15.60:0/390445338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96a4004d40 0x7f96b81f4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:01:59.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.975+0000 7f96bcd04640 1 -- 172.21.15.60:0/390445338 >> 172.21.15.60:0/390445338 conn(0x7f96b80b2f00 msgr2=0x7f96b80cffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:01:59.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.975+0000 7f96bcd04640 1 -- 172.21.15.60:0/390445338 shutdown_connections 2024-05-09T05:01:59.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:01:59.976+0000 7f96bcd04640 1 -- 172.21.15.60:0/390445338 wait complete. 2024-05-09T05:01:59.993 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_extra2 cephfs_data_extra2 --yes-i-really-really-mean-it 2024-05-09T05:02:00.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.105+0000 7fbfee6f8640 1 Processor -- start 2024-05-09T05:02:00.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.105+0000 7fbfee6f8640 1 -- start start 2024-05-09T05:02:00.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.106+0000 7fbfee6f8640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbfe80b01b0 0x7fbfe8144cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:00.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.106+0000 7fbfee6f8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfe80b0590 0x7fbfe81604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:00.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.106+0000 7fbfee6f8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbfe8163200 0x7fbfe8163670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:00.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.106+0000 7fbfee6f8640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fbfe8145ca0 con 0x7fbfe8163200 2024-05-09T05:02:00.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.106+0000 7fbfee6f8640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fbfe8145400 con 0x7fbfe80b01b0 2024-05-09T05:02:00.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.106+0000 7fbfee6f8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbfe81456e0 con 0x7fbfe80b0590 2024-05-09T05:02:00.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.106+0000 7fbfecc95640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbfe8163200 0x7fbfe8163670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:00.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.106+0000 7fbfecc95640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbfe8163200 0x7fbfe8163670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36228/0 (socket says 172.21.15.60:36228) 2024-05-09T05:02:00.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.106+0000 7fbfecc95640 1 -- 172.21.15.60:0/269751111 learned_addr learned my addr 172.21.15.60:0/269751111 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:00.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.106+0000 7fbfe77fe640 1 --2- 172.21.15.60:0/269751111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfe80b0590 0x7fbfe81604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:00.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.106+0000 7fbfe7fff640 1 --2- 172.21.15.60:0/269751111 >> v2:172.21.15.114:3300/0 conn(0x7fbfe80b01b0 0x7fbfe8144cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:00.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.107+0000 7fbfecc95640 1 -- 172.21.15.60:0/269751111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfe80b0590 msgr2=0x7fbfe81604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:00.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.107+0000 7fbfecc95640 1 --2- 172.21.15.60:0/269751111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfe80b0590 0x7fbfe81604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:00.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.107+0000 7fbfecc95640 1 -- 172.21.15.60:0/269751111 >> v2:172.21.15.114:3300/0 conn(0x7fbfe80b01b0 msgr2=0x7fbfe8144cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:00.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.107+0000 7fbfecc95640 1 --2- 172.21.15.60:0/269751111 >> v2:172.21.15.114:3300/0 conn(0x7fbfe80b01b0 0x7fbfe8144cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:00.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.107+0000 7fbfecc95640 1 -- 172.21.15.60:0/269751111 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe81459c0 con 0x7fbfe8163200 2024-05-09T05:02:00.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.107+0000 7fbfe77fe640 1 --2- 172.21.15.60:0/269751111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfe80b0590 0x7fbfe81604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:02:00.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.107+0000 7fbfe7fff640 1 --2- 172.21.15.60:0/269751111 >> v2:172.21.15.114:3300/0 conn(0x7fbfe80b01b0 0x7fbfe8144cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:02:00.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.107+0000 7fbfecc95640 1 --2- 172.21.15.60:0/269751111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfe8163200 0x7fbfe8163670 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8069fb0 tx=0x7fbfe002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d15417fa0a974379 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:00.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.107+0000 7fbfe6ffd640 1 -- 172.21.15.60:0/269751111 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe0036070 con 0x7fbfe8163200 2024-05-09T05:02:00.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.108+0000 7fbfe6ffd640 1 -- 172.21.15.60:0/269751111 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfe0031600 con 0x7fbfe8163200 2024-05-09T05:02:00.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.108+0000 7fbfe6ffd640 1 -- 172.21.15.60:0/269751111 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe00319d0 con 0x7fbfe8163200 2024-05-09T05:02:00.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.108+0000 7fbfee6f8640 1 -- 172.21.15.60:0/269751111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfe8163200 msgr2=0x7fbfe8163670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:00.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.108+0000 7fbfee6f8640 1 --2- 172.21.15.60:0/269751111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfe8163200 0x7fbfe8163670 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8069fb0 tx=0x7fbfe002a6a0 comp rx=0 tx=0).stop 2024-05-09T05:02:00.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.108+0000 7fbfee6f8640 1 -- 172.21.15.60:0/269751111 shutdown_connections 2024-05-09T05:02:00.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.108+0000 7fbfee6f8640 1 --2- 172.21.15.60:0/269751111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfe8163200 0x7fbfe8163670 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:00.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.108+0000 7fbfee6f8640 1 --2- 172.21.15.60:0/269751111 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfe80b0590 0x7fbfe81604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:00.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.108+0000 7fbfee6f8640 1 --2- 172.21.15.60:0/269751111 >> v2:172.21.15.114:3300/0 conn(0x7fbfe80b01b0 0x7fbfe8144cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:00.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.108+0000 7fbfee6f8640 1 -- 172.21.15.60:0/269751111 >> 172.21.15.60:0/269751111 conn(0x7fbfe806f0c0 msgr2=0x7fbfe806f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:00.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.108+0000 7fbfee6f8640 1 -- 172.21.15.60:0/269751111 shutdown_connections 2024-05-09T05:02:00.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.108+0000 7fbfee6f8640 1 -- 172.21.15.60:0/269751111 wait complete. 2024-05-09T05:02:00.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.109+0000 7fbfee6f8640 1 Processor -- start 2024-05-09T05:02:00.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.110+0000 7fbfee6f8640 1 -- start start 2024-05-09T05:02:00.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.110+0000 7fbfee6f8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfe80b01b0 0x7fbfe8140a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:00.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.110+0000 7fbfee6f8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfe80b0590 0x7fbfe8156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:00.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.110+0000 7fbfee6f8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfe8163200 0x7fbfe8156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:00.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.110+0000 7fbfee6f8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe8160ba0 con 0x7fbfe80b0590 2024-05-09T05:02:00.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.110+0000 7fbfee6f8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe8160a20 con 0x7fbfe80b01b0 2024-05-09T05:02:00.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.110+0000 7fbfee6f8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbfe80b09b0 con 0x7fbfe8163200 2024-05-09T05:02:00.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.110+0000 7fbfe77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfe80b0590 0x7fbfe8156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:00.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.111+0000 7fbfe77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfe80b0590 0x7fbfe8156250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36244/0 (socket says 172.21.15.60:36244) 2024-05-09T05:02:00.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.111+0000 7fbfe77fe640 1 -- 172.21.15.60:0/1301356286 learned_addr learned my addr 172.21.15.60:0/1301356286 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:00.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.111+0000 7fbfe7fff640 1 --2- 172.21.15.60:0/1301356286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfe80b01b0 0x7fbfe8140a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:00.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.111+0000 7fbfecc95640 1 --2- 172.21.15.60:0/1301356286 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfe8163200 0x7fbfe8156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:00.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.111+0000 7fbfe77fe640 1 -- 172.21.15.60:0/1301356286 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfe8163200 msgr2=0x7fbfe8156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:00.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.111+0000 7fbfe77fe640 1 --2- 172.21.15.60:0/1301356286 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfe8163200 0x7fbfe8156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:00.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.111+0000 7fbfe77fe640 1 -- 172.21.15.60:0/1301356286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfe80b01b0 msgr2=0x7fbfe8140a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:00.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.111+0000 7fbfe77fe640 1 --2- 172.21.15.60:0/1301356286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfe80b01b0 0x7fbfe8140a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:00.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.111+0000 7fbfe77fe640 1 -- 172.21.15.60:0/1301356286 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe815dc50 con 0x7fbfe80b0590 2024-05-09T05:02:00.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.111+0000 7fbfe77fe640 1 --2- 172.21.15.60:0/1301356286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfe80b0590 0x7fbfe8156250 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00c820 tx=0x7fbfdc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:00.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.111+0000 7fbfe4ff9640 1 -- 172.21.15.60:0/1301356286 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc015070 con 0x7fbfe80b0590 2024-05-09T05:02:00.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.111+0000 7fbfe4ff9640 1 -- 172.21.15.60:0/1301356286 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfdc007a30 con 0x7fbfe80b0590 2024-05-09T05:02:00.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.111+0000 7fbfe4ff9640 1 -- 172.21.15.60:0/1301356286 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc007de0 con 0x7fbfe80b0590 2024-05-09T05:02:00.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.111+0000 7fbfee6f8640 1 -- 172.21.15.60:0/1301356286 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfe815df30 con 0x7fbfe80b0590 2024-05-09T05:02:00.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.111+0000 7fbfee6f8640 1 -- 172.21.15.60:0/1301356286 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfe815e4f0 con 0x7fbfe80b0590 2024-05-09T05:02:00.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.112+0000 7fbfee6f8640 1 -- 172.21.15.60:0/1301356286 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfe8051040 con 0x7fbfe80b0590 2024-05-09T05:02:00.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.113+0000 7fbfe4ff9640 1 -- 172.21.15.60:0/1301356286 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbfdc018070 con 0x7fbfe80b0590 2024-05-09T05:02:00.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.113+0000 7fbfe4ff9640 1 --2- 172.21.15.60:0/1301356286 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbfd4076300 0x7fbfd40787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:00.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.113+0000 7fbfe4ff9640 1 -- 172.21.15.60:0/1301356286 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5740+0+0 (secure 0 0 0) 0x7fbfdc0973a0 con 0x7fbfe80b0590 2024-05-09T05:02:00.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.114+0000 7fbfe7fff640 1 --2- 172.21.15.60:0/1301356286 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbfd4076300 0x7fbfd40787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:00.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.114+0000 7fbfe7fff640 1 --2- 172.21.15.60:0/1301356286 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbfd4076300 0x7fbfd40787c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbfe806dc30 tx=0x7fbfd8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:00.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.118+0000 7fbfe4ff9640 1 -- 172.21.15.60:0/1301356286 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbfdc060720 con 0x7fbfe80b0590 2024-05-09T05:02:00.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.307+0000 7fbfee6f8640 1 -- 172.21.15.60:0/1301356286 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_extra2", "pool2": "cephfs_data_extra2", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbfe8070370 con 0x7fbfe80b0590 2024-05-09T05:02:00.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.979+0000 7fbfe4ff9640 1 -- 172.21.15.60:0/1301356286 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_extra2", "pool2": "cephfs_data_extra2", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_extra2' removed v62) ==== 190+0+0 (secure 0 0 0) 0x7fbfdc064a20 con 0x7fbfe80b0590 2024-05-09T05:02:00.980 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data_extra2' removed 2024-05-09T05:02:00.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.982+0000 7fbfee6f8640 1 -- 172.21.15.60:0/1301356286 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbfd4076300 msgr2=0x7fbfd40787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:00.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.982+0000 7fbfee6f8640 1 --2- 172.21.15.60:0/1301356286 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbfd4076300 0x7fbfd40787c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbfe806dc30 tx=0x7fbfd8008040 comp rx=0 tx=0).stop 2024-05-09T05:02:00.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.982+0000 7fbfee6f8640 1 -- 172.21.15.60:0/1301356286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfe80b0590 msgr2=0x7fbfe8156250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:00.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.982+0000 7fbfee6f8640 1 --2- 172.21.15.60:0/1301356286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfe80b0590 0x7fbfe8156250 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00c820 tx=0x7fbfdc00cce0 comp rx=0 tx=0).stop 2024-05-09T05:02:00.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.982+0000 7fbfee6f8640 1 -- 172.21.15.60:0/1301356286 shutdown_connections 2024-05-09T05:02:00.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.982+0000 7fbfee6f8640 1 --2- 172.21.15.60:0/1301356286 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbfd4076300 0x7fbfd40787c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:00.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.982+0000 7fbfee6f8640 1 --2- 172.21.15.60:0/1301356286 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbfe8163200 0x7fbfe8156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:00.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.982+0000 7fbfee6f8640 1 --2- 172.21.15.60:0/1301356286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfe80b0590 0x7fbfe8156250 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:00.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.982+0000 7fbfee6f8640 1 --2- 172.21.15.60:0/1301356286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfe80b01b0 0x7fbfe8140a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:00.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.982+0000 7fbfee6f8640 1 -- 172.21.15.60:0/1301356286 >> 172.21.15.60:0/1301356286 conn(0x7fbfe806f0c0 msgr2=0x7fbfe8142050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:00.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.983+0000 7fbfee6f8640 1 -- 172.21.15.60:0/1301356286 shutdown_connections 2024-05-09T05:02:00.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:00.983+0000 7fbfee6f8640 1 -- 172.21.15.60:0/1301356286 wait complete. 2024-05-09T05:02:00.997 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-05-09T05:02:01.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.106+0000 7f1a24e45640 1 Processor -- start 2024-05-09T05:02:01.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.106+0000 7f1a24e45640 1 -- start start 2024-05-09T05:02:01.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.107+0000 7f1a24e45640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a200d1050 0x7f1a200ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:01.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.107+0000 7f1a24e45640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1a20096eb0 0x7f1a2009b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:01.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.107+0000 7f1a24e45640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a200af370 con 0x7f1a2009be90 2024-05-09T05:02:01.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.107+0000 7f1a24e45640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f1a20096a00 con 0x7f1a20096eb0 2024-05-09T05:02:01.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.107+0000 7f1a24e45640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1a200d2dd0 con 0x7f1a200d1050 2024-05-09T05:02:01.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.108+0000 7f1a1e59c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a200d1050 0x7f1a200ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:01.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.108+0000 7f1a1dd9b640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1a20096eb0 0x7f1a2009b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:01.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.108+0000 7f1a1e59c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a200d1050 0x7f1a200ae9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.60:60432/0 (socket says 172.21.15.60:60432) 2024-05-09T05:02:01.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.108+0000 7f1a1e59c640 1 -- 172.21.15.60:0/2816359412 learned_addr learned my addr 172.21.15.60:0/2816359412 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:01.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.108+0000 7f1a1e59c640 1 -- 172.21.15.60:0/2816359412 >> v2:172.21.15.114:3300/0 conn(0x7f1a20096eb0 msgr2=0x7f1a2009b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:01.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.108+0000 7f1a1e59c640 1 --2- 172.21.15.60:0/2816359412 >> v2:172.21.15.114:3300/0 conn(0x7f1a20096eb0 0x7f1a2009b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.108+0000 7f1a1e59c640 1 -- 172.21.15.60:0/2816359412 >> v1:172.21.15.114:6789/0 conn(0x7f1a2009be90 legacy=0x7f1a200d1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:01.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.108+0000 7f1a1e59c640 1 -- 172.21.15.60:0/2816359412 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a200d30b0 con 0x7f1a200d1050 2024-05-09T05:02:01.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.109+0000 7f1a1e59c640 1 --2- 172.21.15.60:0/2816359412 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a200d1050 0x7f1a200ae9f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1a18009870 tx=0x7f1a1802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59f41f0bec758056 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:01.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.109+0000 7f1a1d59a640 1 -- 172.21.15.60:0/2816359412 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a18037070 con 0x7f1a200d1050 2024-05-09T05:02:01.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.109+0000 7f1a1d59a640 1 -- 172.21.15.60:0/2816359412 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a18032440 con 0x7f1a200d1050 2024-05-09T05:02:01.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.109+0000 7f1a1d59a640 1 -- 172.21.15.60:0/2816359412 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a18032810 con 0x7f1a200d1050 2024-05-09T05:02:01.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.109+0000 7f1a24e45640 1 -- 172.21.15.60:0/2816359412 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a200d1050 msgr2=0x7f1a200ae9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:01.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.109+0000 7f1a24e45640 1 --2- 172.21.15.60:0/2816359412 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a200d1050 0x7f1a200ae9f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1a18009870 tx=0x7f1a1802a270 comp rx=0 tx=0).stop 2024-05-09T05:02:01.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.110+0000 7f1a24e45640 1 -- 172.21.15.60:0/2816359412 shutdown_connections 2024-05-09T05:02:01.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.110+0000 7f1a24e45640 1 --2- 172.21.15.60:0/2816359412 >> v2:172.21.15.114:3300/0 conn(0x7f1a20096eb0 0x7f1a2009b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.110+0000 7f1a24e45640 1 --2- 172.21.15.60:0/2816359412 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a200d1050 0x7f1a200ae9f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.110+0000 7f1a24e45640 1 -- 172.21.15.60:0/2816359412 >> 172.21.15.60:0/2816359412 conn(0x7f1a200993f0 msgr2=0x7f1a20099800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:01.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.110+0000 7f1a24e45640 1 -- 172.21.15.60:0/2816359412 shutdown_connections 2024-05-09T05:02:01.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.110+0000 7f1a24e45640 1 -- 172.21.15.60:0/2816359412 wait complete. 2024-05-09T05:02:01.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.111+0000 7f1a24e45640 1 Processor -- start 2024-05-09T05:02:01.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.111+0000 7f1a24e45640 1 -- start start 2024-05-09T05:02:01.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.112+0000 7f1a24e45640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a20096eb0 0x7f1a2006e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:01.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.112+0000 7f1a24e45640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a2009be90 0x7f1a201f54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:01.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.112+0000 7f1a24e45640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a200d1050 0x7f1a201f59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:01.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.112+0000 7f1a24e45640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1a200af0b0 con 0x7f1a200d1050 2024-05-09T05:02:01.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.112+0000 7f1a24e45640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1a200aef30 con 0x7f1a2009be90 2024-05-09T05:02:01.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.112+0000 7f1a24e45640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1a200d1b40 con 0x7f1a20096eb0 2024-05-09T05:02:01.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.112+0000 7f1a1ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a200d1050 0x7f1a201f59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:01.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.112+0000 7f1a1ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a200d1050 0x7f1a201f59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36254/0 (socket says 172.21.15.60:36254) 2024-05-09T05:02:01.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.112+0000 7f1a1ed9d640 1 -- 172.21.15.60:0/784306364 learned_addr learned my addr 172.21.15.60:0/784306364 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:01.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.112+0000 7f1a1e59c640 1 --2- 172.21.15.60:0/784306364 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a20096eb0 0x7f1a2006e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:01.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.112+0000 7f1a1dd9b640 1 --2- 172.21.15.60:0/784306364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a2009be90 0x7f1a201f54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:01.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.112+0000 7f1a1ed9d640 1 -- 172.21.15.60:0/784306364 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a20096eb0 msgr2=0x7f1a2006e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:01.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.112+0000 7f1a1ed9d640 1 --2- 172.21.15.60:0/784306364 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a20096eb0 0x7f1a2006e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.112+0000 7f1a1ed9d640 1 -- 172.21.15.60:0/784306364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a2009be90 msgr2=0x7f1a201f54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:01.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.112+0000 7f1a1ed9d640 1 --2- 172.21.15.60:0/784306364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a2009be90 0x7f1a201f54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.112+0000 7f1a1ed9d640 1 -- 172.21.15.60:0/784306364 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a2006ee80 con 0x7f1a200d1050 2024-05-09T05:02:01.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.112+0000 7f1a1ed9d640 1 --2- 172.21.15.60:0/784306364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a200d1050 0x7f1a201f59f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1a1000c7d0 tx=0x7f1a1000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:01.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.113+0000 7f1a16ffd640 1 -- 172.21.15.60:0/784306364 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a10003b70 con 0x7f1a200d1050 2024-05-09T05:02:01.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.113+0000 7f1a16ffd640 1 -- 172.21.15.60:0/784306364 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a10003d30 con 0x7f1a200d1050 2024-05-09T05:02:01.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.113+0000 7f1a24e45640 1 -- 172.21.15.60:0/784306364 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a2006f160 con 0x7f1a200d1050 2024-05-09T05:02:01.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.113+0000 7f1a16ffd640 1 -- 172.21.15.60:0/784306364 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a1000aa90 con 0x7f1a200d1050 2024-05-09T05:02:01.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.113+0000 7f1a24e45640 1 -- 172.21.15.60:0/784306364 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a2006f720 con 0x7f1a200d1050 2024-05-09T05:02:01.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.114+0000 7f1a24e45640 1 -- 172.21.15.60:0/784306364 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a20075540 con 0x7f1a200d1050 2024-05-09T05:02:01.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.114+0000 7f1a16ffd640 1 -- 172.21.15.60:0/784306364 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1a1000ac30 con 0x7f1a200d1050 2024-05-09T05:02:01.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.115+0000 7f1a16ffd640 1 --2- 172.21.15.60:0/784306364 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1a000762d0 0x7f1a00078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:01.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.115+0000 7f1a16ffd640 1 -- 172.21.15.60:0/784306364 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7f1a100971d0 con 0x7f1a200d1050 2024-05-09T05:02:01.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.115+0000 7f1a1e59c640 1 --2- 172.21.15.60:0/784306364 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1a000762d0 0x7f1a00078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:01.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.116+0000 7f1a1e59c640 1 --2- 172.21.15.60:0/784306364 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1a000762d0 0x7f1a00078790 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1a18034000 tx=0x7f1a1802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:01.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.120+0000 7f1a16ffd640 1 -- 172.21.15.60:0/784306364 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1a10060650 con 0x7f1a200d1050 2024-05-09T05:02:01.315 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.314+0000 7f1a24e45640 1 -- 172.21.15.60:0/784306364 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f1a20051040 con 0x7f1a200d1050 2024-05-09T05:02:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.317+0000 7f1a16ffd640 1 -- 172.21.15.60:0/784306364 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v16) ==== 104+0+0 (secure 0 0 0) 0x7f1a10064950 con 0x7f1a200d1050 2024-05-09T05:02:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.320+0000 7f1a24e45640 1 -- 172.21.15.60:0/784306364 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1a000762d0 msgr2=0x7f1a00078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.320+0000 7f1a24e45640 1 --2- 172.21.15.60:0/784306364 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1a000762d0 0x7f1a00078790 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1a18034000 tx=0x7f1a1802e000 comp rx=0 tx=0).stop 2024-05-09T05:02:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.320+0000 7f1a24e45640 1 -- 172.21.15.60:0/784306364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a200d1050 msgr2=0x7f1a201f59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.320+0000 7f1a24e45640 1 --2- 172.21.15.60:0/784306364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a200d1050 0x7f1a201f59f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1a1000c7d0 tx=0x7f1a1000cc90 comp rx=0 tx=0).stop 2024-05-09T05:02:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.320+0000 7f1a24e45640 1 -- 172.21.15.60:0/784306364 shutdown_connections 2024-05-09T05:02:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.320+0000 7f1a24e45640 1 --2- 172.21.15.60:0/784306364 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1a000762d0 0x7f1a00078790 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.320+0000 7f1a24e45640 1 --2- 172.21.15.60:0/784306364 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a200d1050 0x7f1a201f59f0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.320+0000 7f1a24e45640 1 --2- 172.21.15.60:0/784306364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a2009be90 0x7f1a201f54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.320+0000 7f1a24e45640 1 --2- 172.21.15.60:0/784306364 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a20096eb0 0x7f1a2006e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.320+0000 7f1a24e45640 1 -- 172.21.15.60:0/784306364 >> 172.21.15.60:0/784306364 conn(0x7f1a200993f0 msgr2=0x7f1a200997d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.320+0000 7f1a24e45640 1 -- 172.21.15.60:0/784306364 shutdown_connections 2024-05-09T05:02:01.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.320+0000 7f1a24e45640 1 -- 172.21.15.60:0/784306364 wait complete. 2024-05-09T05:02:01.332 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_backtrace.TestBacktrace.test_backtrace_flush_on_deleted_data_pool' 2024-05-09T05:02:01.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.435+0000 7fc56f9c8640 1 Processor -- start 2024-05-09T05:02:01.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.435+0000 7fc56f9c8640 1 -- start start 2024-05-09T05:02:01.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.436+0000 7fc56f9c8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc568159930 0x7fc5681601b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:01.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.436+0000 7fc56f9c8640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc568072130 0x7fc5680725b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:01.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.436+0000 7fc56f9c8640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fc568162eb0 0x7fc568163340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:01.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.436+0000 7fc56f9c8640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fc568159d10 con 0x7fc568072130 2024-05-09T05:02:01.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.436+0000 7fc56f9c8640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fc56806f8e0 con 0x7fc568162eb0 2024-05-09T05:02:01.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.436+0000 7fc56f9c8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc56806fbc0 con 0x7fc568159930 2024-05-09T05:02:01.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.436+0000 7fc56cf63640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc568072130 0x7fc5680725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:01.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.436+0000 7fc56cf63640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc568072130 0x7fc5680725b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36258/0 (socket says 172.21.15.60:36258) 2024-05-09T05:02:01.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.436+0000 7fc56cf63640 1 -- 172.21.15.60:0/2013297624 learned_addr learned my addr 172.21.15.60:0/2013297624 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:01.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.436+0000 7fc56d764640 1 --2- 172.21.15.60:0/2013297624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc568159930 0x7fc5681601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:01.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.436+0000 7fc56df65640 1 --2- 172.21.15.60:0/2013297624 >> v2:172.21.15.114:3300/0 conn(0x7fc568162eb0 0x7fc568163340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:01.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.436+0000 7fc56cf63640 1 -- 172.21.15.60:0/2013297624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc568159930 msgr2=0x7fc5681601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:01.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.436+0000 7fc56cf63640 1 --2- 172.21.15.60:0/2013297624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc568159930 0x7fc5681601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.436+0000 7fc56cf63640 1 -- 172.21.15.60:0/2013297624 >> v2:172.21.15.114:3300/0 conn(0x7fc568162eb0 msgr2=0x7fc568163340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:01.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.436+0000 7fc56cf63640 1 --2- 172.21.15.60:0/2013297624 >> v2:172.21.15.114:3300/0 conn(0x7fc568162eb0 0x7fc568163340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.436+0000 7fc56cf63640 1 -- 172.21.15.60:0/2013297624 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56806fea0 con 0x7fc568072130 2024-05-09T05:02:01.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.437+0000 7fc56cf63640 1 --2- 172.21.15.60:0/2013297624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc568072130 0x7fc5680725b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc56806f870 tx=0x7fc55802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed6773f0103f1907 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:01.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.437+0000 7fc55ffff640 1 -- 172.21.15.60:0/2013297624 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc558037070 con 0x7fc568072130 2024-05-09T05:02:01.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.437+0000 7fc55ffff640 1 -- 172.21.15.60:0/2013297624 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc558032440 con 0x7fc568072130 2024-05-09T05:02:01.439 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.437+0000 7fc55ffff640 1 -- 172.21.15.60:0/2013297624 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc558032810 con 0x7fc568072130 2024-05-09T05:02:01.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.437+0000 7fc56f9c8640 1 -- 172.21.15.60:0/2013297624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc568072130 msgr2=0x7fc5680725b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:01.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.437+0000 7fc56f9c8640 1 --2- 172.21.15.60:0/2013297624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc568072130 0x7fc5680725b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc56806f870 tx=0x7fc55802a1e0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.438+0000 7fc56f9c8640 1 -- 172.21.15.60:0/2013297624 shutdown_connections 2024-05-09T05:02:01.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.438+0000 7fc56f9c8640 1 --2- 172.21.15.60:0/2013297624 >> v2:172.21.15.114:3300/0 conn(0x7fc568162eb0 0x7fc568163340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.438+0000 7fc56f9c8640 1 --2- 172.21.15.60:0/2013297624 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc568072130 0x7fc5680725b0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.438+0000 7fc56f9c8640 1 --2- 172.21.15.60:0/2013297624 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc568159930 0x7fc5681601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.438+0000 7fc56f9c8640 1 -- 172.21.15.60:0/2013297624 >> 172.21.15.60:0/2013297624 conn(0x7fc5680b4c30 msgr2=0x7fc568073240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:01.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.438+0000 7fc56f9c8640 1 -- 172.21.15.60:0/2013297624 shutdown_connections 2024-05-09T05:02:01.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.438+0000 7fc56f9c8640 1 -- 172.21.15.60:0/2013297624 wait complete. 2024-05-09T05:02:01.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.439+0000 7fc56f9c8640 1 Processor -- start 2024-05-09T05:02:01.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.439+0000 7fc56f9c8640 1 -- start start 2024-05-09T05:02:01.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc56f9c8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc568072130 0x7fc568159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:01.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc56f9c8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc568159930 0x7fc5680ad190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:01.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc56f9c8640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc568162eb0 0x7fc568151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:01.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc56f9c8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fc5681606f0 con 0x7fc568072130 2024-05-09T05:02:01.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc56f9c8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc568070460 con 0x7fc568159930 2024-05-09T05:02:01.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc56f9c8640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fc56815fae0 con 0x7fc568162eb0 2024-05-09T05:02:01.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc56d764640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc568072130 0x7fc568159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:01.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc56d764640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc568072130 0x7fc568159ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36270/0 (socket says 172.21.15.60:36270) 2024-05-09T05:02:01.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc56d764640 1 -- 172.21.15.60:0/505637148 learned_addr learned my addr 172.21.15.60:0/505637148 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:01.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc56cf63640 1 --2- 172.21.15.60:0/505637148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc568159930 0x7fc5680ad190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:01.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc56df65640 1 --2- 172.21.15.60:0/505637148 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc568162eb0 0x7fc568151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:01.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc56d764640 1 -- 172.21.15.60:0/505637148 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc568162eb0 msgr2=0x7fc568151a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:01.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc56d764640 1 --2- 172.21.15.60:0/505637148 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc568162eb0 0x7fc568151a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc56d764640 1 -- 172.21.15.60:0/505637148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc568159930 msgr2=0x7fc5680ad190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:01.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc56d764640 1 --2- 172.21.15.60:0/505637148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc568159930 0x7fc5680ad190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc56d764640 1 -- 172.21.15.60:0/505637148 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5680ac000 con 0x7fc568072130 2024-05-09T05:02:01.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc56d764640 1 --2- 172.21.15.60:0/505637148 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc568072130 0x7fc568159ef0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fc56400e890 tx=0x7fc56400ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:01.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc55dffb640 1 -- 172.21.15.60:0/505637148 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc564017070 con 0x7fc568072130 2024-05-09T05:02:01.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc55dffb640 1 -- 172.21.15.60:0/505637148 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc56400c940 con 0x7fc568072130 2024-05-09T05:02:01.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.440+0000 7fc56f9c8640 1 -- 172.21.15.60:0/505637148 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5680ac2e0 con 0x7fc568072130 2024-05-09T05:02:01.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.441+0000 7fc55dffb640 1 -- 172.21.15.60:0/505637148 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56400cd10 con 0x7fc568072130 2024-05-09T05:02:01.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.441+0000 7fc56f9c8640 1 -- 172.21.15.60:0/505637148 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5680acc70 con 0x7fc568072130 2024-05-09T05:02:01.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.442+0000 7fc56f9c8640 1 -- 172.21.15.60:0/505637148 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc568051040 con 0x7fc568072130 2024-05-09T05:02:01.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.442+0000 7fc55dffb640 1 -- 172.21.15.60:0/505637148 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc56401a070 con 0x7fc568072130 2024-05-09T05:02:01.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.443+0000 7fc55dffb640 1 --2- 172.21.15.60:0/505637148 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc54c0762d0 0x7fc54c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:01.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.443+0000 7fc55dffb640 1 -- 172.21.15.60:0/505637148 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7fc564097400 con 0x7fc568072130 2024-05-09T05:02:01.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.443+0000 7fc56cf63640 1 --2- 172.21.15.60:0/505637148 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc54c0762d0 0x7fc54c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:01.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.444+0000 7fc56cf63640 1 --2- 172.21.15.60:0/505637148 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc54c0762d0 0x7fc54c078790 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc558034000 tx=0x7fc55802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:01.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.448+0000 7fc55dffb640 1 -- 172.21.15.60:0/505637148 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc564060870 con 0x7fc568072130 2024-05-09T05:02:01.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.712+0000 7fc56f9c8640 1 -- 172.21.15.60:0/505637148 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_backtrace.TestBacktrace.test_backtrace_flush_on_deleted_data_pool"]} v 0) -- 0x7fc56806cc70 con 0x7fc568072130 2024-05-09T05:02:01.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.980+0000 7fc55dffb640 1 -- 172.21.15.60:0/505637148 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_backtrace.TestBacktrace.test_backtrace_flush_on_deleted_data_pool"]}]=0 v118) ==== 162+0+0 (secure 0 0 0) 0x7fc564064b70 con 0x7fc568072130 2024-05-09T05:02:01.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.983+0000 7fc56f9c8640 1 -- 172.21.15.60:0/505637148 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc54c0762d0 msgr2=0x7fc54c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:01.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.983+0000 7fc56f9c8640 1 --2- 172.21.15.60:0/505637148 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc54c0762d0 0x7fc54c078790 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc558034000 tx=0x7fc55802e000 comp rx=0 tx=0).stop 2024-05-09T05:02:01.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.983+0000 7fc56f9c8640 1 -- 172.21.15.60:0/505637148 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc568072130 msgr2=0x7fc568159ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:01.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.983+0000 7fc56f9c8640 1 --2- 172.21.15.60:0/505637148 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc568072130 0x7fc568159ef0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fc56400e890 tx=0x7fc56400ed50 comp rx=0 tx=0).stop 2024-05-09T05:02:01.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.983+0000 7fc56f9c8640 1 -- 172.21.15.60:0/505637148 shutdown_connections 2024-05-09T05:02:01.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.983+0000 7fc56f9c8640 1 --2- 172.21.15.60:0/505637148 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fc54c0762d0 0x7fc54c078790 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.983+0000 7fc56f9c8640 1 --2- 172.21.15.60:0/505637148 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fc568162eb0 0x7fc568151a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.983+0000 7fc56f9c8640 1 --2- 172.21.15.60:0/505637148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc568159930 0x7fc5680ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.983+0000 7fc56f9c8640 1 --2- 172.21.15.60:0/505637148 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc568072130 0x7fc568159ef0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:01.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.983+0000 7fc56f9c8640 1 -- 172.21.15.60:0/505637148 >> 172.21.15.60:0/505637148 conn(0x7fc5680b4c30 msgr2=0x7fc56815e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:01.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.983+0000 7fc56f9c8640 1 -- 172.21.15.60:0/505637148 shutdown_connections 2024-05-09T05:02:01.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:01.983+0000 7fc56f9c8640 1 -- 172.21.15.60:0/505637148 wait complete. 2024-05-09T05:02:01.997 INFO:tasks.cephfs_test_runner:that the MDS does not go read-only when handling backtrace update errors ... ERROR 2024-05-09T05:02:01.997 INFO:tasks.cephfs_test_runner: 2024-05-09T05:02:01.998 INFO:tasks.cephfs_test_runner:====================================================================== 2024-05-09T05:02:01.998 INFO:tasks.cephfs_test_runner:ERROR: test_backtrace_flush_on_deleted_data_pool (tasks.cephfs.test_backtrace.TestBacktrace) 2024-05-09T05:02:01.998 INFO:tasks.cephfs_test_runner:that the MDS does not go read-only when handling backtrace update errors 2024-05-09T05:02:01.998 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-05-09T05:02:01.998 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-05-09T05:02:01.998 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_vshankar_ceph_a2e1088fa282408acd6a8a29f9417b763c0a8663/qa/tasks/cephfs/test_backtrace.py", line 121, in test_backtrace_flush_on_deleted_data_pool 2024-05-09T05:02:01.998 INFO:tasks.cephfs_test_runner: self.mount_a.run_shell(["setfattr", "-x", "ceph.dir.layout.pool", "dir_x"]) 2024-05-09T05:02:01.998 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_vshankar_ceph_a2e1088fa282408acd6a8a29f9417b763c0a8663/qa/tasks/cephfs/mount.py", line 776, in run_shell 2024-05-09T05:02:01.998 INFO:tasks.cephfs_test_runner: return self.client_remote.run(args=args, **kwargs) 2024-05-09T05:02:01.998 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run 2024-05-09T05:02:01.998 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-05-09T05:02:01.998 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run 2024-05-09T05:02:01.998 INFO:tasks.cephfs_test_runner: r.wait() 2024-05-09T05:02:01.998 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait 2024-05-09T05:02:01.998 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-05-09T05:02:01.999 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-05-09T05:02:01.999 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-05-09T05:02:01.999 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 1: '(cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -x ceph.dir.layout.pool dir_x)' 2024-05-09T05:02:01.999 INFO:tasks.cephfs_test_runner: 2024-05-09T05:02:01.999 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-05-09T05:02:01.999 INFO:tasks.cephfs_test_runner:Ran 2 tests in 75.473s 2024-05-09T05:02:01.999 INFO:tasks.cephfs_test_runner: 2024-05-09T05:02:01.999 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-05-09T05:02:01.999 INFO:tasks.cephfs_test_runner: 2024-05-09T05:02:01.999 INFO:tasks.cephfs_test_runner:====================================================================== 2024-05-09T05:02:01.999 INFO:tasks.cephfs_test_runner:ERROR: test_backtrace_flush_on_deleted_data_pool (tasks.cephfs.test_backtrace.TestBacktrace) 2024-05-09T05:02:01.999 INFO:tasks.cephfs_test_runner:that the MDS does not go read-only when handling backtrace update errors 2024-05-09T05:02:01.999 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-05-09T05:02:01.999 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-05-09T05:02:02.000 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_vshankar_ceph_a2e1088fa282408acd6a8a29f9417b763c0a8663/qa/tasks/cephfs/test_backtrace.py", line 121, in test_backtrace_flush_on_deleted_data_pool 2024-05-09T05:02:02.000 INFO:tasks.cephfs_test_runner: self.mount_a.run_shell(["setfattr", "-x", "ceph.dir.layout.pool", "dir_x"]) 2024-05-09T05:02:02.000 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_vshankar_ceph_a2e1088fa282408acd6a8a29f9417b763c0a8663/qa/tasks/cephfs/mount.py", line 776, in run_shell 2024-05-09T05:02:02.000 INFO:tasks.cephfs_test_runner: return self.client_remote.run(args=args, **kwargs) 2024-05-09T05:02:02.000 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run 2024-05-09T05:02:02.000 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-05-09T05:02:02.000 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run 2024-05-09T05:02:02.000 INFO:tasks.cephfs_test_runner: r.wait() 2024-05-09T05:02:02.000 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait 2024-05-09T05:02:02.000 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-05-09T05:02:02.000 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-05-09T05:02:02.000 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-05-09T05:02:02.000 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 1: '(cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -x ceph.dir.layout.pool dir_x)' 2024-05-09T05:02:02.000 INFO:tasks.cephfs_test_runner: 2024-05-09T05:02:02.001 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_vshankar_ceph_a2e1088fa282408acd6a8a29f9417b763c0a8663/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_backtrace_flush_on_deleted_data_pool (tasks.cephfs.test_backtrace.TestBacktrace) 2024-05-09T05:02:02.253 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=cc1110b8798e4127896dc1850e77f79c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_vshankar_ceph_a2e1088fa282408acd6a8a29f9417b763c0a8663/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_backtrace_flush_on_deleted_data_pool (tasks.cephfs.test_backtrace.TestBacktrace) 2024-05-09T05:02:02.254 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-05-09T05:02:02.263 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-05-09T05:02:02.311 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-09T05:02:02.311 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:02:02.311 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:02:02.333 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057102,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498908k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19678 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498908k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T05:02:02.333 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-05-09T05:02:02.333 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:02:02.392 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057102,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498908k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19678 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498908k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T05:02:02.392 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T05:02:02.392 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:02:02.412 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056322,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497656k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248824k,nr_inodes=812206,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497656k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T05:02:02.412 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-05-09T05:02:02.412 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:02:02.471 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056322,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497656k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248824k,nr_inodes=812206,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497656k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T05:02:02.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:02:02.471 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2024-05-09T05:02:02.490 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T05:02:02.490 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T05:02:02.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:02:02.491 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T05:02:02.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:02:02.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T05:02:02.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:02:02.631 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link delete ceph-brx 2024-05-09T05:02:02.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:02:02.731 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2024-05-09T05:02:02.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:02:02.748 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link delete ceph-brx 2024-05-09T05:02:02.818 INFO:teuthology.orchestra.run.smithi060.stderr:Cannot find device "ceph-brx" 2024-05-09T05:02:02.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T05:02:02.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:02:02.820 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-09T05:02:02.839 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-05-09T05:02:02.839 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-05-09T05:02:02.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:02:02.840 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-09T05:02:02.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:02:02.911 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T05:02:02.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:02:02.981 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2024-05-09T05:02:03.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:02:03.073 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-05-09T05:02:03.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:02:03.089 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2024-05-09T05:02:03.158 INFO:teuthology.orchestra.run.smithi114.stderr:Cannot find device "ceph-brx" 2024-05-09T05:02:03.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T05:02:03.159 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-09T05:02:03.169 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-09T05:02:03.169 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:02:03.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.276+0000 7f929199e640 1 Processor -- start 2024-05-09T05:02:03.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.276+0000 7f929199e640 1 -- start start 2024-05-09T05:02:03.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.277+0000 7f929199e640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f928c098d30 0x7f928c15fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:03.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.277+0000 7f929199e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f928c0991c0 con 0x7f928c15c000 2024-05-09T05:02:03.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.277+0000 7f929199e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f928c09abd0 con 0x7f928c160330 2024-05-09T05:02:03.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.277+0000 7f929199e640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f928c0d0a80 con 0x7f928c098d30 2024-05-09T05:02:03.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.277+0000 7f928bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f928c098d30 0x7f928c15fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:03.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.277+0000 7f929099c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f928c15c000 0x7f928c074220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34248/0 (socket says 172.21.15.60:34248) 2024-05-09T05:02:03.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.277+0000 7f929099c640 1 -- 172.21.15.60:0/3677893976 learned_addr learned my addr 172.21.15.60:0/3677893976 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:02:03.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.277+0000 7f928bfff640 1 -- 172.21.15.60:0/3677893976 >> v1:172.21.15.114:6789/0 conn(0x7f928c160330 legacy=0x7f928c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:03.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.277+0000 7f928bfff640 1 -- 172.21.15.60:0/3677893976 >> v1:172.21.15.60:6789/0 conn(0x7f928c15c000 legacy=0x7f928c074220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:03.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.277+0000 7f928bfff640 1 -- 172.21.15.60:0/3677893976 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f928c09aeb0 con 0x7f928c098d30 2024-05-09T05:02:03.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.278+0000 7f928bfff640 1 --2- 172.21.15.60:0/3677893976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f928c098d30 0x7f928c15fdf0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f92740097a0 tx=0x7f927402a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e52816efc1597d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:03.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.278+0000 7f928b7fe640 1 -- 172.21.15.60:0/3677893976 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9274037070 con 0x7f928c098d30 2024-05-09T05:02:03.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.278+0000 7f928b7fe640 1 -- 172.21.15.60:0/3677893976 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f927402ad10 con 0x7f928c098d30 2024-05-09T05:02:03.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.278+0000 7f928b7fe640 1 -- 172.21.15.60:0/3677893976 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9274032660 con 0x7f928c098d30 2024-05-09T05:02:03.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.279+0000 7f929199e640 1 -- 172.21.15.60:0/3677893976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f928c098d30 msgr2=0x7f928c15fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:03.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.279+0000 7f929199e640 1 --2- 172.21.15.60:0/3677893976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f928c098d30 0x7f928c15fdf0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f92740097a0 tx=0x7f927402a080 comp rx=0 tx=0).stop 2024-05-09T05:02:03.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.279+0000 7f929199e640 1 -- 172.21.15.60:0/3677893976 shutdown_connections 2024-05-09T05:02:03.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.279+0000 7f929199e640 1 --2- 172.21.15.60:0/3677893976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f928c098d30 0x7f928c15fdf0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.279+0000 7f929199e640 1 -- 172.21.15.60:0/3677893976 >> 172.21.15.60:0/3677893976 conn(0x7f928c0b41f0 msgr2=0x7f928c0b45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:03.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.279+0000 7f929199e640 1 -- 172.21.15.60:0/3677893976 shutdown_connections 2024-05-09T05:02:03.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.279+0000 7f929199e640 1 -- 172.21.15.60:0/3677893976 wait complete. 2024-05-09T05:02:03.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.280+0000 7f929199e640 1 Processor -- start 2024-05-09T05:02:03.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.280+0000 7f929199e640 1 -- start start 2024-05-09T05:02:03.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.281+0000 7f929199e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f928c15c000 0x7f928c1f5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:03.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.281+0000 7f929199e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f928c160330 0x7f928c1f7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:03.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.281+0000 7f929199e640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f928c0adc50 0x7f928c0ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:03.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.281+0000 7f929199e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f928c0994a0 con 0x7f928c15c000 2024-05-09T05:02:03.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.281+0000 7f929199e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f928c070af0 con 0x7f928c160330 2024-05-09T05:02:03.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.281+0000 7f929199e640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f928c070c70 con 0x7f928c0adc50 2024-05-09T05:02:03.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.281+0000 7f929099c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f928c15c000 0x7f928c1f5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:03.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.281+0000 7f929099c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f928c15c000 0x7f928c1f5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36290/0 (socket says 172.21.15.60:36290) 2024-05-09T05:02:03.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.281+0000 7f929099c640 1 -- 172.21.15.60:0/3007705409 learned_addr learned my addr 172.21.15.60:0/3007705409 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:03.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.282+0000 7f928bfff640 1 --2- 172.21.15.60:0/3007705409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f928c160330 0x7f928c1f7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:03.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.282+0000 7f929119d640 1 --2- 172.21.15.60:0/3007705409 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f928c0adc50 0x7f928c0ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:03.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.282+0000 7f929099c640 1 -- 172.21.15.60:0/3007705409 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f928c0adc50 msgr2=0x7f928c0ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:03.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.282+0000 7f929099c640 1 --2- 172.21.15.60:0/3007705409 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f928c0adc50 0x7f928c0ae100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.282+0000 7f929099c640 1 -- 172.21.15.60:0/3007705409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f928c160330 msgr2=0x7f928c1f7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:03.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.282+0000 7f929099c640 1 --2- 172.21.15.60:0/3007705409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f928c160330 0x7f928c1f7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.282+0000 7f929099c640 1 -- 172.21.15.60:0/3007705409 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f928c159790 con 0x7f928c15c000 2024-05-09T05:02:03.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.282+0000 7f928bfff640 1 --2- 172.21.15.60:0/3007705409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f928c160330 0x7f928c1f7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:02:03.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.282+0000 7f929099c640 1 --2- 172.21.15.60:0/3007705409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f928c15c000 0x7f928c1f5280 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f927c00c760 tx=0x7f927c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:03.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.282+0000 7f92897fa640 1 -- 172.21.15.60:0/3007705409 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f927c018070 con 0x7f928c15c000 2024-05-09T05:02:03.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.282+0000 7f92897fa640 1 -- 172.21.15.60:0/3007705409 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f927c0072f0 con 0x7f928c15c000 2024-05-09T05:02:03.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.282+0000 7f929199e640 1 -- 172.21.15.60:0/3007705409 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f928c159a70 con 0x7f928c15c000 2024-05-09T05:02:03.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.282+0000 7f92897fa640 1 -- 172.21.15.60:0/3007705409 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f927c0047e0 con 0x7f928c15c000 2024-05-09T05:02:03.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.282+0000 7f929199e640 1 -- 172.21.15.60:0/3007705409 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f928c0afdb0 con 0x7f928c15c000 2024-05-09T05:02:03.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.283+0000 7f929199e640 1 -- 172.21.15.60:0/3007705409 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f928c15fb50 con 0x7f928c15c000 2024-05-09T05:02:03.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.284+0000 7f92897fa640 1 -- 172.21.15.60:0/3007705409 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f927c01b070 con 0x7f928c15c000 2024-05-09T05:02:03.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.284+0000 7f92897fa640 1 --2- 172.21.15.60:0/3007705409 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f926c0762d0 0x7f926c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:03.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.285+0000 7f92897fa640 1 -- 172.21.15.60:0/3007705409 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7f927c096900 con 0x7f928c15c000 2024-05-09T05:02:03.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.285+0000 7f928bfff640 1 --2- 172.21.15.60:0/3007705409 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f926c0762d0 0x7f926c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:03.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.285+0000 7f928bfff640 1 --2- 172.21.15.60:0/3007705409 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f926c0762d0 0x7f926c078790 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9274034000 tx=0x7f927402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:03.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.289+0000 7f92897fa640 1 -- 172.21.15.60:0/3007705409 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f928c15fb50 con 0x7f928c15c000 2024-05-09T05:02:03.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.475+0000 7f929199e640 1 -- 172.21.15.60:0/3007705409 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f928c0640c0 con 0x7f926c0762d0 2024-05-09T05:02:03.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.476+0000 7f92897fa640 1 -- 172.21.15.60:0/3007705409 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13423 (secure 0 0 0) 0x7f928c0640c0 con 0x7f926c0762d0 2024-05-09T05:02:03.477 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:02:03.477 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-05-09T05:02:03.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.478+0000 7f929199e640 1 -- 172.21.15.60:0/3007705409 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f926c0762d0 msgr2=0x7f926c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.478+0000 7f929199e640 1 --2- 172.21.15.60:0/3007705409 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f926c0762d0 0x7f926c078790 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9274034000 tx=0x7f927402e000 comp rx=0 tx=0).stop 2024-05-09T05:02:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.478+0000 7f929199e640 1 -- 172.21.15.60:0/3007705409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f928c15c000 msgr2=0x7f928c1f5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.478+0000 7f929199e640 1 --2- 172.21.15.60:0/3007705409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f928c15c000 0x7f928c1f5280 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f927c00c760 tx=0x7f927c00cc20 comp rx=0 tx=0).stop 2024-05-09T05:02:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.478+0000 7f929199e640 1 -- 172.21.15.60:0/3007705409 shutdown_connections 2024-05-09T05:02:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.478+0000 7f929199e640 1 --2- 172.21.15.60:0/3007705409 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f926c0762d0 0x7f926c078790 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.478+0000 7f929199e640 1 --2- 172.21.15.60:0/3007705409 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f928c0adc50 0x7f928c0ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.478+0000 7f929199e640 1 --2- 172.21.15.60:0/3007705409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f928c160330 0x7f928c1f7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.478+0000 7f929199e640 1 --2- 172.21.15.60:0/3007705409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f928c15c000 0x7f928c1f5280 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.478+0000 7f929199e640 1 -- 172.21.15.60:0/3007705409 >> 172.21.15.60:0/3007705409 conn(0x7f928c0b41f0 msgr2=0x7f928c0a9550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:03.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.478+0000 7f929199e640 1 -- 172.21.15.60:0/3007705409 shutdown_connections 2024-05-09T05:02:03.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.478+0000 7f929199e640 1 -- 172.21.15.60:0/3007705409 wait complete. 2024-05-09T05:02:03.490 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-05-09T05:02:02.934828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":206,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":239856,"kb_used_data":7032,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734864,"statfs":{"total":773094113280,"available":772848500736,"internally_reserved":0,"allocated":7200768,"data_stored":3939380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-2,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.968041"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":188,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-09T05:02:01.660678+0000","last_change":"2024-05-09T05:00:08.667158+0000","last_active":"2024-05-09T05:02:01.660678+0000","last_peered":"2024-05-09T05:02:01.660678+0000","last_clean":"2024-05-09T05:02:01.660678+0000","last_became_active":"2024-05-09T05:00:08.666316+0000","last_became_peered":"2024-05-09T05:00:08.666316+0000","last_unstale":"2024-05-09T05:02:01.660678+0000","last_undegraded":"2024-05-09T05:02:01.660678+0000","last_fullsized":"2024-05-09T05:02:01.660678+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_clean_scrub_stamp":"2024-05-09T05:00:07.370119+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:20.234643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28344,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343496,"statfs":{"total":96636764160,"available":96607739904,"internally_reserved":0,"allocated":1355776,"data_stored":938916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":745472,"data_stored":342600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":745472,"data_stored":342600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27768,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":761856,"data_stored":348548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":761856,"data_stored":348548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31912,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339928,"statfs":{"total":96636764160,"available":96604086272,"internally_reserved":0,"allocated":745472,"data_stored":342600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":745472,"data_stored":342600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1339392,"data_stored":932968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:02:03.490 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:02:03.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.594+0000 7f18d55c1640 1 Processor -- start 2024-05-09T05:02:03.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.594+0000 7f18d55c1640 1 -- start start 2024-05-09T05:02:03.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.594+0000 7f18d55c1640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f18d015f300 0x7f18d01639f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:03.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.594+0000 7f18d55c1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18d01606f0 0x7f18d006ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:03.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.595+0000 7f18d55c1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18d006ff60 con 0x7f18d006f9f0 2024-05-09T05:02:03.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.595+0000 7f18d55c1640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f18d0070240 con 0x7f18d015f300 2024-05-09T05:02:03.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.595+0000 7f18d55c1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f18d0070520 con 0x7f18d01606f0 2024-05-09T05:02:03.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.595+0000 7f18ceffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f18d015f300 0x7f18d01639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:03.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.595+0000 7f18ceffd640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f18d015f300 0x7f18d01639f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36304/0 (socket says 172.21.15.60:36304) 2024-05-09T05:02:03.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.595+0000 7f18ceffd640 1 -- 172.21.15.60:0/3715595955 learned_addr learned my addr 172.21.15.60:0/3715595955 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:03.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.595+0000 7f18cf7fe640 1 --2- 172.21.15.60:0/3715595955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18d01606f0 0x7f18d006ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:03.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.595+0000 7f18ceffd640 1 -- 172.21.15.60:0/3715595955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18d01606f0 msgr2=0x7f18d006ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:03.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.595+0000 7f18ceffd640 1 --2- 172.21.15.60:0/3715595955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18d01606f0 0x7f18d006ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.595+0000 7f18ceffd640 1 -- 172.21.15.60:0/3715595955 >> v1:172.21.15.114:6789/0 conn(0x7f18d006f9f0 legacy=0x7f18d00d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:03.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.595+0000 7f18ceffd640 1 -- 172.21.15.60:0/3715595955 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d006e740 con 0x7f18d015f300 2024-05-09T05:02:03.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.596+0000 7f18ceffd640 1 --2- 172.21.15.60:0/3715595955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18d015f300 0x7f18d01639f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f18c0009870 tx=0x7f18c002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6ab314867b72105 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:03.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.596+0000 7f18ce7fc640 1 -- 172.21.15.60:0/3715595955 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c0037070 con 0x7f18d015f300 2024-05-09T05:02:03.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.596+0000 7f18ce7fc640 1 -- 172.21.15.60:0/3715595955 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18c0032440 con 0x7f18d015f300 2024-05-09T05:02:03.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.596+0000 7f18ce7fc640 1 -- 172.21.15.60:0/3715595955 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c0032810 con 0x7f18d015f300 2024-05-09T05:02:03.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.596+0000 7f18d55c1640 1 -- 172.21.15.60:0/3715595955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18d015f300 msgr2=0x7f18d01639f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:03.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.596+0000 7f18d55c1640 1 --2- 172.21.15.60:0/3715595955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18d015f300 0x7f18d01639f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f18c0009870 tx=0x7f18c002a1c0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.597+0000 7f18d55c1640 1 -- 172.21.15.60:0/3715595955 shutdown_connections 2024-05-09T05:02:03.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.597+0000 7f18d55c1640 1 --2- 172.21.15.60:0/3715595955 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18d01606f0 0x7f18d006ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.597+0000 7f18d55c1640 1 --2- 172.21.15.60:0/3715595955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18d015f300 0x7f18d01639f0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.597+0000 7f18d55c1640 1 -- 172.21.15.60:0/3715595955 >> 172.21.15.60:0/3715595955 conn(0x7f18d009b850 msgr2=0x7f18d009bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:03.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.597+0000 7f18d55c1640 1 -- 172.21.15.60:0/3715595955 shutdown_connections 2024-05-09T05:02:03.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.597+0000 7f18d55c1640 1 -- 172.21.15.60:0/3715595955 wait complete. 2024-05-09T05:02:03.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.598+0000 7f18d55c1640 1 Processor -- start 2024-05-09T05:02:03.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.598+0000 7f18d55c1640 1 -- start start 2024-05-09T05:02:03.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.599+0000 7f18d55c1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18d006f9f0 0x7f18d015deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:03.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.599+0000 7f18d55c1640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18d015f300 0x7f18d015b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:03.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.599+0000 7f18d55c1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18d01606f0 0x7f18d00af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:03.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.599+0000 7f18d55c1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f18d00d3b00 con 0x7f18d01606f0 2024-05-09T05:02:03.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.599+0000 7f18d55c1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f18d0163f30 con 0x7f18d006f9f0 2024-05-09T05:02:03.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.599+0000 7f18d55c1640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f18d00d3c80 con 0x7f18d015f300 2024-05-09T05:02:03.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.599+0000 7f18cf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18d01606f0 0x7f18d00af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:03.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.599+0000 7f18cf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18d01606f0 0x7f18d00af9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36320/0 (socket says 172.21.15.60:36320) 2024-05-09T05:02:03.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.599+0000 7f18cf7fe640 1 -- 172.21.15.60:0/3622544883 learned_addr learned my addr 172.21.15.60:0/3622544883 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:03.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.599+0000 7f18ceffd640 1 --2- 172.21.15.60:0/3622544883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18d006f9f0 0x7f18d015deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:03.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.599+0000 7f18c67fc640 1 --2- 172.21.15.60:0/3622544883 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18d015f300 0x7f18d015b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:03.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.599+0000 7f18cf7fe640 1 -- 172.21.15.60:0/3622544883 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18d015f300 msgr2=0x7f18d015b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:03.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.599+0000 7f18cf7fe640 1 --2- 172.21.15.60:0/3622544883 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18d015f300 0x7f18d015b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.599+0000 7f18cf7fe640 1 -- 172.21.15.60:0/3622544883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18d006f9f0 msgr2=0x7f18d015deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:03.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.599+0000 7f18cf7fe640 1 --2- 172.21.15.60:0/3622544883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18d006f9f0 0x7f18d015deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.599+0000 7f18cf7fe640 1 -- 172.21.15.60:0/3622544883 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d01f92a0 con 0x7f18d01606f0 2024-05-09T05:02:03.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.599+0000 7f18cf7fe640 1 --2- 172.21.15.60:0/3622544883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18d01606f0 0x7f18d00af9a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f18c8007e90 tx=0x7f18c800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:03.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.600+0000 7f18c7fff640 1 -- 172.21.15.60:0/3622544883 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c8014420 con 0x7f18d01606f0 2024-05-09T05:02:03.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.600+0000 7f18c7fff640 1 -- 172.21.15.60:0/3622544883 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18c8019070 con 0x7f18d01606f0 2024-05-09T05:02:03.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.600+0000 7f18d55c1640 1 -- 172.21.15.60:0/3622544883 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18d00ad6b0 con 0x7f18d01606f0 2024-05-09T05:02:03.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.600+0000 7f18c7fff640 1 -- 172.21.15.60:0/3622544883 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c8014800 con 0x7f18d01606f0 2024-05-09T05:02:03.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.600+0000 7f18d55c1640 1 -- 172.21.15.60:0/3622544883 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18d00adc70 con 0x7f18d01606f0 2024-05-09T05:02:03.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.601+0000 7f18d55c1640 1 -- 172.21.15.60:0/3622544883 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18d00ade70 con 0x7f18d01606f0 2024-05-09T05:02:03.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.601+0000 7f18c7fff640 1 -- 172.21.15.60:0/3622544883 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f18c80149a0 con 0x7f18d01606f0 2024-05-09T05:02:03.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.602+0000 7f18c7fff640 1 --2- 172.21.15.60:0/3622544883 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f18b00762d0 0x7f18b0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:03.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.602+0000 7f18c7fff640 1 -- 172.21.15.60:0/3622544883 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7f18c8097b30 con 0x7f18d01606f0 2024-05-09T05:02:03.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.602+0000 7f18ceffd640 1 --2- 172.21.15.60:0/3622544883 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f18b00762d0 0x7f18b0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:03.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.602+0000 7f18ceffd640 1 --2- 172.21.15.60:0/3622544883 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f18b00762d0 0x7f18b0078790 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f18c0034000 tx=0x7f18c002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:03.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.606+0000 7f18c7fff640 1 -- 172.21.15.60:0/3622544883 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f18d00ade70 con 0x7f18d01606f0 2024-05-09T05:02:03.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.794+0000 7f18d55c1640 1 -- 172.21.15.60:0/3622544883 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f18d0051040 con 0x7f18b00762d0 2024-05-09T05:02:03.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.795+0000 7f18c7fff640 1 -- 172.21.15.60:0/3622544883 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13423 (secure 0 0 0) 0x7f18d0051040 con 0x7f18b00762d0 2024-05-09T05:02:03.796 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:02:03.796 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-05-09T05:02:03.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.797+0000 7f18d55c1640 1 -- 172.21.15.60:0/3622544883 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f18b00762d0 msgr2=0x7f18b0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:03.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.797+0000 7f18d55c1640 1 --2- 172.21.15.60:0/3622544883 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f18b00762d0 0x7f18b0078790 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f18c0034000 tx=0x7f18c002e000 comp rx=0 tx=0).stop 2024-05-09T05:02:03.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.797+0000 7f18d55c1640 1 -- 172.21.15.60:0/3622544883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18d01606f0 msgr2=0x7f18d00af9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:03.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.797+0000 7f18d55c1640 1 --2- 172.21.15.60:0/3622544883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18d01606f0 0x7f18d00af9a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f18c8007e90 tx=0x7f18c800e4b0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.797+0000 7f18d55c1640 1 -- 172.21.15.60:0/3622544883 shutdown_connections 2024-05-09T05:02:03.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.797+0000 7f18d55c1640 1 --2- 172.21.15.60:0/3622544883 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f18b00762d0 0x7f18b0078790 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.797+0000 7f18d55c1640 1 --2- 172.21.15.60:0/3622544883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f18d01606f0 0x7f18d00af9a0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.797+0000 7f18d55c1640 1 --2- 172.21.15.60:0/3622544883 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f18d015f300 0x7f18d015b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.797+0000 7f18d55c1640 1 --2- 172.21.15.60:0/3622544883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18d006f9f0 0x7f18d015deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.797+0000 7f18d55c1640 1 -- 172.21.15.60:0/3622544883 >> 172.21.15.60:0/3622544883 conn(0x7f18d009b850 msgr2=0x7f18d0162620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:03.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.797+0000 7f18d55c1640 1 -- 172.21.15.60:0/3622544883 shutdown_connections 2024-05-09T05:02:03.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.797+0000 7f18d55c1640 1 -- 172.21.15.60:0/3622544883 wait complete. 2024-05-09T05:02:03.809 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-05-09T05:02:02.934828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":206,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":239856,"kb_used_data":7032,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734864,"statfs":{"total":773094113280,"available":772848500736,"internally_reserved":0,"allocated":7200768,"data_stored":3939380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-2,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.968041"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":188,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-09T05:02:01.660678+0000","last_change":"2024-05-09T05:00:08.667158+0000","last_active":"2024-05-09T05:02:01.660678+0000","last_peered":"2024-05-09T05:02:01.660678+0000","last_clean":"2024-05-09T05:02:01.660678+0000","last_became_active":"2024-05-09T05:00:08.666316+0000","last_became_peered":"2024-05-09T05:00:08.666316+0000","last_unstale":"2024-05-09T05:02:01.660678+0000","last_undegraded":"2024-05-09T05:02:01.660678+0000","last_fullsized":"2024-05-09T05:02:01.660678+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_clean_scrub_stamp":"2024-05-09T05:00:07.370119+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:20.234643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28344,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343496,"statfs":{"total":96636764160,"available":96607739904,"internally_reserved":0,"allocated":1355776,"data_stored":938916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":745472,"data_stored":342600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":745472,"data_stored":342600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27768,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":761856,"data_stored":348548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":761856,"data_stored":348548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31912,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339928,"statfs":{"total":96636764160,"available":96604086272,"internally_reserved":0,"allocated":745472,"data_stored":342600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":745472,"data_stored":342600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1339392,"data_stored":932968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:02:03.810 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-09T05:02:03.810 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:02:03.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.914+0000 7f05c9054640 1 Processor -- start 2024-05-09T05:02:03.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.914+0000 7f05c9054640 1 -- start start 2024-05-09T05:02:03.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.915+0000 7f05c9054640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f05c40ae1f0 0x7f05c409d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:03.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.915+0000 7f05c9054640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f05c40b2da0 0x7f05c40b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:03.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.915+0000 7f05c9054640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05c409c7f0 con 0x7f05c40b1040 2024-05-09T05:02:03.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.915+0000 7f05c9054640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f05c409cdb0 con 0x7f05c40ae1f0 2024-05-09T05:02:03.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.915+0000 7f05c9054640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f05c409cad0 con 0x7f05c40b2da0 2024-05-09T05:02:03.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.915+0000 7f05c2d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f05c40ae1f0 0x7f05c409d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:03.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.915+0000 7f05c2d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f05c40ae1f0 0x7f05c409d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36322/0 (socket says 172.21.15.60:36322) 2024-05-09T05:02:03.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.915+0000 7f05c2d9d640 1 -- 172.21.15.60:0/2655143354 learned_addr learned my addr 172.21.15.60:0/2655143354 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:03.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.915+0000 7f05c359e640 1 --2- 172.21.15.60:0/2655143354 >> v2:172.21.15.114:3300/0 conn(0x7f05c40b2da0 0x7f05c40b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:03.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.915+0000 7f05c2d9d640 1 -- 172.21.15.60:0/2655143354 >> v2:172.21.15.114:3300/0 conn(0x7f05c40b2da0 msgr2=0x7f05c40b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:03.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.915+0000 7f05c2d9d640 1 --2- 172.21.15.60:0/2655143354 >> v2:172.21.15.114:3300/0 conn(0x7f05c40b2da0 0x7f05c40b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.915+0000 7f05c2d9d640 1 -- 172.21.15.60:0/2655143354 >> v1:172.21.15.114:6789/0 conn(0x7f05c40b1040 legacy=0x7f05c40b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:03.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.915+0000 7f05c2d9d640 1 -- 172.21.15.60:0/2655143354 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05c40b25a0 con 0x7f05c40ae1f0 2024-05-09T05:02:03.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.916+0000 7f05c2d9d640 1 --2- 172.21.15.60:0/2655143354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05c40ae1f0 0x7f05c409d180 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f05c4070a90 tx=0x7f05b402a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2063e7e67cc33fd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:03.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.916+0000 7f05c259c640 1 -- 172.21.15.60:0/2655143354 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b4037070 con 0x7f05c40ae1f0 2024-05-09T05:02:03.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.916+0000 7f05c259c640 1 -- 172.21.15.60:0/2655143354 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05b402ae00 con 0x7f05c40ae1f0 2024-05-09T05:02:03.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.916+0000 7f05c259c640 1 -- 172.21.15.60:0/2655143354 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b4032660 con 0x7f05c40ae1f0 2024-05-09T05:02:03.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.917+0000 7f05c9054640 1 -- 172.21.15.60:0/2655143354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05c40ae1f0 msgr2=0x7f05c409d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:03.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.917+0000 7f05c9054640 1 --2- 172.21.15.60:0/2655143354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05c40ae1f0 0x7f05c409d180 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f05c4070a90 tx=0x7f05b402a0b0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.917+0000 7f05c9054640 1 -- 172.21.15.60:0/2655143354 shutdown_connections 2024-05-09T05:02:03.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.917+0000 7f05c9054640 1 --2- 172.21.15.60:0/2655143354 >> v2:172.21.15.114:3300/0 conn(0x7f05c40b2da0 0x7f05c40b31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.917+0000 7f05c9054640 1 --2- 172.21.15.60:0/2655143354 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05c40ae1f0 0x7f05c409d180 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.917+0000 7f05c9054640 1 -- 172.21.15.60:0/2655143354 >> 172.21.15.60:0/2655143354 conn(0x7f05c4072390 msgr2=0x7f05c40727a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:03.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.917+0000 7f05c9054640 1 -- 172.21.15.60:0/2655143354 shutdown_connections 2024-05-09T05:02:03.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.917+0000 7f05c9054640 1 -- 172.21.15.60:0/2655143354 wait complete. 2024-05-09T05:02:03.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.918+0000 7f05c9054640 1 Processor -- start 2024-05-09T05:02:03.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.918+0000 7f05c9054640 1 -- start start 2024-05-09T05:02:03.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.919+0000 7f05c9054640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05c40ae1f0 0x7f05c41f8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:03.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.919+0000 7f05c9054640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05c40b1040 0x7f05c41f4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:03.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.919+0000 7f05c9054640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05c40b2da0 0x7f05c4162190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:03.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.919+0000 7f05c9054640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f05c409b6b0 con 0x7f05c40b2da0 2024-05-09T05:02:03.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.919+0000 7f05c9054640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f05c40ab4b0 con 0x7f05c40ae1f0 2024-05-09T05:02:03.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.919+0000 7f05c9054640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f05c409b830 con 0x7f05c40b1040 2024-05-09T05:02:03.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.919+0000 7f05c359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05c40b2da0 0x7f05c4162190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:03.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.919+0000 7f05c359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05c40b2da0 0x7f05c4162190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36330/0 (socket says 172.21.15.60:36330) 2024-05-09T05:02:03.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.919+0000 7f05c359e640 1 -- 172.21.15.60:0/2065794493 learned_addr learned my addr 172.21.15.60:0/2065794493 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:03.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.920+0000 7f05c2d9d640 1 --2- 172.21.15.60:0/2065794493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05c40ae1f0 0x7f05c41f8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:03.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.920+0000 7f05bbfff640 1 --2- 172.21.15.60:0/2065794493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05c40b1040 0x7f05c41f4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:03.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.920+0000 7f05c359e640 1 -- 172.21.15.60:0/2065794493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05c40b1040 msgr2=0x7f05c41f4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:03.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.920+0000 7f05c359e640 1 --2- 172.21.15.60:0/2065794493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05c40b1040 0x7f05c41f4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.920+0000 7f05c359e640 1 -- 172.21.15.60:0/2065794493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05c40ae1f0 msgr2=0x7f05c41f8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:03.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.920+0000 7f05c359e640 1 --2- 172.21.15.60:0/2065794493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05c40ae1f0 0x7f05c41f8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:03.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.920+0000 7f05c359e640 1 -- 172.21.15.60:0/2065794493 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05c41580f0 con 0x7f05c40b2da0 2024-05-09T05:02:03.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.920+0000 7f05bbfff640 1 --2- 172.21.15.60:0/2065794493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05c40b1040 0x7f05c41f4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:02:03.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.920+0000 7f05c2d9d640 1 --2- 172.21.15.60:0/2065794493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05c40ae1f0 0x7f05c41f8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:02:03.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.920+0000 7f05c359e640 1 --2- 172.21.15.60:0/2065794493 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05c40b2da0 0x7f05c4162190 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f05bc00cc30 tx=0x7f05bc00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:03.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.920+0000 7f05bb7fe640 1 -- 172.21.15.60:0/2065794493 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05bc00ce40 con 0x7f05c40b2da0 2024-05-09T05:02:03.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.920+0000 7f05bb7fe640 1 -- 172.21.15.60:0/2065794493 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05bc003990 con 0x7f05c40b2da0 2024-05-09T05:02:03.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.920+0000 7f05c9054640 1 -- 172.21.15.60:0/2065794493 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05c41f5670 con 0x7f05c40b2da0 2024-05-09T05:02:03.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.920+0000 7f05bb7fe640 1 -- 172.21.15.60:0/2065794493 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05bc003d40 con 0x7f05c40b2da0 2024-05-09T05:02:03.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.920+0000 7f05c9054640 1 -- 172.21.15.60:0/2065794493 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05c41f5c30 con 0x7f05c40b2da0 2024-05-09T05:02:03.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.921+0000 7f05c9054640 1 -- 172.21.15.60:0/2065794493 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05c4097100 con 0x7f05c40b2da0 2024-05-09T05:02:03.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.922+0000 7f05bb7fe640 1 -- 172.21.15.60:0/2065794493 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f05bc004000 con 0x7f05c40b2da0 2024-05-09T05:02:03.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.922+0000 7f05bb7fe640 1 --2- 172.21.15.60:0/2065794493 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f05a40762d0 0x7f05a4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:03.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.922+0000 7f05bb7fe640 1 -- 172.21.15.60:0/2065794493 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7f05bc0978f0 con 0x7f05c40b2da0 2024-05-09T05:02:03.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.922+0000 7f05c2d9d640 1 --2- 172.21.15.60:0/2065794493 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f05a40762d0 0x7f05a4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:03.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.923+0000 7f05c2d9d640 1 --2- 172.21.15.60:0/2065794493 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f05a40762d0 0x7f05a4078790 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f05b4034000 tx=0x7f05b402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:03.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:03.927+0000 7f05bb7fe640 1 -- 172.21.15.60:0/2065794493 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f05bc014030 con 0x7f05c40b2da0 2024-05-09T05:02:04.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.115+0000 7f05c9054640 1 -- 172.21.15.60:0/2065794493 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f05c4096e20 con 0x7f05a40762d0 2024-05-09T05:02:04.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.117+0000 7f05bb7fe640 1 -- 172.21.15.60:0/2065794493 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13423 (secure 0 0 0) 0x7f05c4096e20 con 0x7f05a40762d0 2024-05-09T05:02:04.117 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:02:04.118 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-05-09T05:02:04.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.119+0000 7f05c9054640 1 -- 172.21.15.60:0/2065794493 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f05a40762d0 msgr2=0x7f05a4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:04.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.119+0000 7f05c9054640 1 --2- 172.21.15.60:0/2065794493 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f05a40762d0 0x7f05a4078790 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f05b4034000 tx=0x7f05b402e000 comp rx=0 tx=0).stop 2024-05-09T05:02:04.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.119+0000 7f05c9054640 1 -- 172.21.15.60:0/2065794493 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05c40b2da0 msgr2=0x7f05c4162190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:04.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.119+0000 7f05c9054640 1 --2- 172.21.15.60:0/2065794493 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05c40b2da0 0x7f05c4162190 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f05bc00cc30 tx=0x7f05bc00a5c0 comp rx=0 tx=0).stop 2024-05-09T05:02:04.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.119+0000 7f05c9054640 1 -- 172.21.15.60:0/2065794493 shutdown_connections 2024-05-09T05:02:04.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.119+0000 7f05c9054640 1 --2- 172.21.15.60:0/2065794493 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f05a40762d0 0x7f05a4078790 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:04.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.119+0000 7f05c9054640 1 --2- 172.21.15.60:0/2065794493 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05c40b2da0 0x7f05c4162190 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:04.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.119+0000 7f05c9054640 1 --2- 172.21.15.60:0/2065794493 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f05c40b1040 0x7f05c41f4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:04.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.119+0000 7f05c9054640 1 --2- 172.21.15.60:0/2065794493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f05c40ae1f0 0x7f05c41f8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:04.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.119+0000 7f05c9054640 1 -- 172.21.15.60:0/2065794493 >> 172.21.15.60:0/2065794493 conn(0x7f05c4072390 msgr2=0x7f05c409b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:04.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.119+0000 7f05c9054640 1 -- 172.21.15.60:0/2065794493 shutdown_connections 2024-05-09T05:02:04.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.119+0000 7f05c9054640 1 -- 172.21.15.60:0/2065794493 wait complete. 2024-05-09T05:02:04.130 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-05-09T05:02:02.934828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":206,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":239856,"kb_used_data":7032,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734864,"statfs":{"total":773094113280,"available":772848500736,"internally_reserved":0,"allocated":7200768,"data_stored":3939380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-2,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.968041"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":188,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-09T05:02:01.660678+0000","last_change":"2024-05-09T05:00:08.667158+0000","last_active":"2024-05-09T05:02:01.660678+0000","last_peered":"2024-05-09T05:02:01.660678+0000","last_clean":"2024-05-09T05:02:01.660678+0000","last_became_active":"2024-05-09T05:00:08.666316+0000","last_became_peered":"2024-05-09T05:00:08.666316+0000","last_unstale":"2024-05-09T05:02:01.660678+0000","last_undegraded":"2024-05-09T05:02:01.660678+0000","last_fullsized":"2024-05-09T05:02:01.660678+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_clean_scrub_stamp":"2024-05-09T05:00:07.370119+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:20.234643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28344,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343496,"statfs":{"total":96636764160,"available":96607739904,"internally_reserved":0,"allocated":1355776,"data_stored":938916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":745472,"data_stored":342600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":745472,"data_stored":342600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27768,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":761856,"data_stored":348548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":761856,"data_stored":348548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31912,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339928,"statfs":{"total":96636764160,"available":96604086272,"internally_reserved":0,"allocated":745472,"data_stored":342600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":745472,"data_stored":342600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1339392,"data_stored":932968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:02:04.131 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:02:04.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.234+0000 7fbd7ca74640 1 Processor -- start 2024-05-09T05:02:04.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.234+0000 7fbd7ca74640 1 -- start start 2024-05-09T05:02:04.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.235+0000 7fbd7ca74640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fbd7815c3a0 0x7fbd78070150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:04.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.235+0000 7fbd7ca74640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbd780ac430 0x7fbd78070690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:04.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.235+0000 7fbd7ca74640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd78071b80 con 0x7fbd78097de0 2024-05-09T05:02:04.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.235+0000 7fbd7ca74640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fbd78071e60 con 0x7fbd780ac430 2024-05-09T05:02:04.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.235+0000 7fbd7ca74640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fbd78072140 con 0x7fbd7815c3a0 2024-05-09T05:02:04.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.235+0000 7fbd6ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbd780ac430 0x7fbd78070690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:04.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.235+0000 7fbd6ffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbd780ac430 0x7fbd78070690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36340/0 (socket says 172.21.15.60:36340) 2024-05-09T05:02:04.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.235+0000 7fbd6ffff640 1 -- 172.21.15.60:0/2082694387 learned_addr learned my addr 172.21.15.60:0/2082694387 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:04.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.235+0000 7fbd6ffff640 1 -- 172.21.15.60:0/2082694387 >> v2:172.21.15.114:3300/0 conn(0x7fbd7815c3a0 msgr2=0x7fbd78070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:04.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.235+0000 7fbd6ffff640 1 --2- 172.21.15.60:0/2082694387 >> v2:172.21.15.114:3300/0 conn(0x7fbd7815c3a0 0x7fbd78070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:04.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.235+0000 7fbd6ffff640 1 -- 172.21.15.60:0/2082694387 >> v1:172.21.15.114:6789/0 conn(0x7fbd78097de0 legacy=0x7fbd78099a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:04.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.235+0000 7fbd6ffff640 1 -- 172.21.15.60:0/2082694387 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd78072420 con 0x7fbd780ac430 2024-05-09T05:02:04.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.236+0000 7fbd6ffff640 1 --2- 172.21.15.60:0/2082694387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd780ac430 0x7fbd78070690 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fbd780b1fc0 tx=0x7fbd6002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1637ada169f22da0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:04.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.236+0000 7fbd75d9b640 1 -- 172.21.15.60:0/2082694387 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd60037070 con 0x7fbd780ac430 2024-05-09T05:02:04.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.236+0000 7fbd75d9b640 1 -- 172.21.15.60:0/2082694387 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd60032440 con 0x7fbd780ac430 2024-05-09T05:02:04.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.236+0000 7fbd75d9b640 1 -- 172.21.15.60:0/2082694387 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd60032810 con 0x7fbd780ac430 2024-05-09T05:02:04.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.236+0000 7fbd7ca74640 1 -- 172.21.15.60:0/2082694387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd780ac430 msgr2=0x7fbd78070690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:04.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.236+0000 7fbd7ca74640 1 --2- 172.21.15.60:0/2082694387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd780ac430 0x7fbd78070690 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fbd780b1fc0 tx=0x7fbd6002a1e0 comp rx=0 tx=0).stop 2024-05-09T05:02:04.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.237+0000 7fbd7ca74640 1 -- 172.21.15.60:0/2082694387 shutdown_connections 2024-05-09T05:02:04.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.237+0000 7fbd7ca74640 1 --2- 172.21.15.60:0/2082694387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd780ac430 0x7fbd78070690 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:04.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.237+0000 7fbd7ca74640 1 --2- 172.21.15.60:0/2082694387 >> v2:172.21.15.114:3300/0 conn(0x7fbd7815c3a0 0x7fbd78070150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:04.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.237+0000 7fbd7ca74640 1 -- 172.21.15.60:0/2082694387 >> 172.21.15.60:0/2082694387 conn(0x7fbd780972f0 msgr2=0x7fbd780976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:04.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.237+0000 7fbd7ca74640 1 -- 172.21.15.60:0/2082694387 shutdown_connections 2024-05-09T05:02:04.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.237+0000 7fbd7ca74640 1 -- 172.21.15.60:0/2082694387 wait complete. 2024-05-09T05:02:04.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.238+0000 7fbd7ca74640 1 Processor -- start 2024-05-09T05:02:04.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.238+0000 7fbd7ca74640 1 -- start start 2024-05-09T05:02:04.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.239+0000 7fbd7ca74640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd78097de0 0x7fbd781e6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:04.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.239+0000 7fbd7ca74640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd780ac430 0x7fbd7815dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:04.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.239+0000 7fbd7ca74640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd7815c3a0 0x7fbd78156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:04.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.239+0000 7fbd7ca74640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fbd78071810 con 0x7fbd78097de0 2024-05-09T05:02:04.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.239+0000 7fbd7ca74640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7815c780 con 0x7fbd780ac430 2024-05-09T05:02:04.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.239+0000 7fbd7ca74640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fbd78073c30 con 0x7fbd7815c3a0 2024-05-09T05:02:04.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.239+0000 7fbd7659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd78097de0 0x7fbd781e6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:04.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.239+0000 7fbd7659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd78097de0 0x7fbd781e6b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36350/0 (socket says 172.21.15.60:36350) 2024-05-09T05:02:04.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.239+0000 7fbd7659c640 1 -- 172.21.15.60:0/2219821576 learned_addr learned my addr 172.21.15.60:0/2219821576 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:04.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.239+0000 7fbd6ffff640 1 --2- 172.21.15.60:0/2219821576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd780ac430 0x7fbd7815dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:04.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.239+0000 7fbd76d9d640 1 --2- 172.21.15.60:0/2219821576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd7815c3a0 0x7fbd78156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:04.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.239+0000 7fbd7659c640 1 -- 172.21.15.60:0/2219821576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd7815c3a0 msgr2=0x7fbd78156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:04.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.239+0000 7fbd7659c640 1 --2- 172.21.15.60:0/2219821576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd7815c3a0 0x7fbd78156460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:04.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.239+0000 7fbd7659c640 1 -- 172.21.15.60:0/2219821576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd780ac430 msgr2=0x7fbd7815dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:04.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.239+0000 7fbd7659c640 1 --2- 172.21.15.60:0/2219821576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd780ac430 0x7fbd7815dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:04.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.239+0000 7fbd7659c640 1 -- 172.21.15.60:0/2219821576 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd78096560 con 0x7fbd78097de0 2024-05-09T05:02:04.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.239+0000 7fbd7659c640 1 --2- 172.21.15.60:0/2219821576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd78097de0 0x7fbd781e6b80 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fbd6800b410 tx=0x7fbd6800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:04.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.240+0000 7fbd6effd640 1 -- 172.21.15.60:0/2219821576 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd68015020 con 0x7fbd78097de0 2024-05-09T05:02:04.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.240+0000 7fbd6effd640 1 -- 172.21.15.60:0/2219821576 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd68011850 con 0x7fbd78097de0 2024-05-09T05:02:04.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.240+0000 7fbd7ca74640 1 -- 172.21.15.60:0/2219821576 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd78096840 con 0x7fbd78097de0 2024-05-09T05:02:04.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.240+0000 7fbd6effd640 1 -- 172.21.15.60:0/2219821576 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd68011c40 con 0x7fbd78097de0 2024-05-09T05:02:04.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.240+0000 7fbd7ca74640 1 -- 172.21.15.60:0/2219821576 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd7815c9a0 con 0x7fbd78097de0 2024-05-09T05:02:04.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.241+0000 7fbd7ca74640 1 -- 172.21.15.60:0/2219821576 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd78051040 con 0x7fbd78097de0 2024-05-09T05:02:04.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.241+0000 7fbd6effd640 1 -- 172.21.15.60:0/2219821576 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbd68018020 con 0x7fbd78097de0 2024-05-09T05:02:04.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.242+0000 7fbd6effd640 1 --2- 172.21.15.60:0/2219821576 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbd580762d0 0x7fbd58078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:04.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.242+0000 7fbd6effd640 1 -- 172.21.15.60:0/2219821576 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7fbd68097300 con 0x7fbd78097de0 2024-05-09T05:02:04.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.242+0000 7fbd6ffff640 1 --2- 172.21.15.60:0/2219821576 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbd580762d0 0x7fbd58078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:04.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.243+0000 7fbd6ffff640 1 --2- 172.21.15.60:0/2219821576 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbd580762d0 0x7fbd58078790 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbd60034000 tx=0x7fbd6002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:04.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.247+0000 7fbd6effd640 1 -- 172.21.15.60:0/2219821576 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbd6801f020 con 0x7fbd78097de0 2024-05-09T05:02:04.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.439+0000 7fbd7ca74640 1 -- 172.21.15.60:0/2219821576 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbd7809a7c0 con 0x7fbd78097de0 2024-05-09T05:02:04.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.440+0000 7fbd6effd640 1 -- 172.21.15.60:0/2219821576 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) ==== 74+0+13270 (secure 0 0 0) 0x7fbd68060dd0 con 0x7fbd78097de0 2024-05-09T05:02:04.442 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:02:04.442 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":62,"fsid":"0b1eb66f-9fff-4f82-a7bb-0ccafa2e73b2","created":"2024-05-09T04:59:56.483938+0000","modified":"2024-05-09T05:02:00.976435+0000","last_up_change":"2024-05-09T05:00:06.360267+0000","last_in_change":"2024-05-09T04:59:58.591160+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:00:06.925911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bccd2765-9913-4f63-8661-ad7f5337d356","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6801","nonce":1298086532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6805","nonce":1298086532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6813","nonce":1298086532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":1298086532},{"type":"v1","addr":"172.21.15.60:6809","nonce":1298086532}]},"public_addr":"172.21.15.60:6801/1298086532","cluster_addr":"172.21.15.60:6805/1298086532","heartbeat_back_addr":"172.21.15.60:6813/1298086532","heartbeat_front_addr":"172.21.15.60:6809/1298086532","state":["exists","up"]},{"osd":1,"uuid":"a32ec935-149d-4a8e-a59e-3417d4a61475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6804","nonce":291519275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6808","nonce":291519275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6815","nonce":291519275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":291519275},{"type":"v1","addr":"172.21.15.60:6812","nonce":291519275}]},"public_addr":"172.21.15.60:6804/291519275","cluster_addr":"172.21.15.60:6808/291519275","heartbeat_back_addr":"172.21.15.60:6815/291519275","heartbeat_front_addr":"172.21.15.60:6812/291519275","state":["exists","up"]},{"osd":2,"uuid":"0277ac75-0fad-42ea-b2ad-07c34457aa03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6817","nonce":483347218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6819","nonce":483347218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6823","nonce":483347218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":483347218},{"type":"v1","addr":"172.21.15.60:6821","nonce":483347218}]},"public_addr":"172.21.15.60:6817/483347218","cluster_addr":"172.21.15.60:6819/483347218","heartbeat_back_addr":"172.21.15.60:6823/483347218","heartbeat_front_addr":"172.21.15.60:6821/483347218","state":["exists","up"]},{"osd":3,"uuid":"0978499e-d158-435a-8623-18c8133c31ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6825","nonce":1069806621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6827","nonce":1069806621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6831","nonce":1069806621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1069806621},{"type":"v1","addr":"172.21.15.60:6829","nonce":1069806621}]},"public_addr":"172.21.15.60:6825/1069806621","cluster_addr":"172.21.15.60:6827/1069806621","heartbeat_back_addr":"172.21.15.60:6831/1069806621","heartbeat_front_addr":"172.21.15.60:6829/1069806621","state":["exists","up"]},{"osd":4,"uuid":"484c05ba-6ed7-4417-b5e4-b8dceda47ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6803","nonce":3462725972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6807","nonce":3462725972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6813","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6815","nonce":3462725972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6809","nonce":3462725972},{"type":"v1","addr":"172.21.15.114:6811","nonce":3462725972}]},"public_addr":"172.21.15.114:6803/3462725972","cluster_addr":"172.21.15.114:6807/3462725972","heartbeat_back_addr":"172.21.15.114:6815/3462725972","heartbeat_front_addr":"172.21.15.114:6811/3462725972","state":["exists","up"]},{"osd":5,"uuid":"42c3b489-393d-4bbe-9045-06609caac26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6817","nonce":292113074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6819","nonce":292113074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6823","nonce":292113074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":292113074},{"type":"v1","addr":"172.21.15.114:6821","nonce":292113074}]},"public_addr":"172.21.15.114:6817/292113074","cluster_addr":"172.21.15.114:6819/292113074","heartbeat_back_addr":"172.21.15.114:6823/292113074","heartbeat_front_addr":"172.21.15.114:6821/292113074","state":["exists","up"]},{"osd":6,"uuid":"335cac6a-28ba-449c-9f6d-721790845461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6825","nonce":2612522523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6827","nonce":2612522523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6831","nonce":2612522523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2612522523},{"type":"v1","addr":"172.21.15.114:6829","nonce":2612522523}]},"public_addr":"172.21.15.114:6825/2612522523","cluster_addr":"172.21.15.114:6827/2612522523","heartbeat_back_addr":"172.21.15.114:6831/2612522523","heartbeat_front_addr":"172.21.15.114:6829/2612522523","state":["exists","up"]},{"osd":7,"uuid":"8ed03228-ebba-492d-9d72-936b27450f14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6802","nonce":4142062928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6806","nonce":4142062928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6814","nonce":4142062928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4142062928},{"type":"v1","addr":"172.21.15.114:6810","nonce":4142062928}]},"public_addr":"172.21.15.114:6802/4142062928","cluster_addr":"172.21.15.114:6806/4142062928","heartbeat_back_addr":"172.21.15.114:6814/4142062928","heartbeat_front_addr":"172.21.15.114:6810/4142062928","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.697140+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.634962+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.711049+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.741800+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.768597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.726687+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.790199+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:00:03.701408+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:6835/2425263408":"2024-05-10T05:01:55.921867+0000","172.21.15.60:6833/2425263408":"2024-05-10T05:01:55.921867+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:02:04.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.442+0000 7fbd7ca74640 1 -- 172.21.15.60:0/2219821576 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbd580762d0 msgr2=0x7fbd58078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:04.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.442+0000 7fbd7ca74640 1 --2- 172.21.15.60:0/2219821576 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbd580762d0 0x7fbd58078790 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbd60034000 tx=0x7fbd6002e000 comp rx=0 tx=0).stop 2024-05-09T05:02:04.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.442+0000 7fbd7ca74640 1 -- 172.21.15.60:0/2219821576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd78097de0 msgr2=0x7fbd781e6b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:04.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.442+0000 7fbd7ca74640 1 --2- 172.21.15.60:0/2219821576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd78097de0 0x7fbd781e6b80 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fbd6800b410 tx=0x7fbd6800b8d0 comp rx=0 tx=0).stop 2024-05-09T05:02:04.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.442+0000 7fbd7ca74640 1 -- 172.21.15.60:0/2219821576 shutdown_connections 2024-05-09T05:02:04.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.442+0000 7fbd7ca74640 1 --2- 172.21.15.60:0/2219821576 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fbd580762d0 0x7fbd58078790 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:04.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.442+0000 7fbd7ca74640 1 --2- 172.21.15.60:0/2219821576 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fbd7815c3a0 0x7fbd78156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:04.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.442+0000 7fbd7ca74640 1 --2- 172.21.15.60:0/2219821576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd780ac430 0x7fbd7815dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:04.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.442+0000 7fbd7ca74640 1 --2- 172.21.15.60:0/2219821576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbd78097de0 0x7fbd781e6b80 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:04.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.442+0000 7fbd7ca74640 1 -- 172.21.15.60:0/2219821576 >> 172.21.15.60:0/2219821576 conn(0x7fbd780972f0 msgr2=0x7fbd7815aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:04.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.442+0000 7fbd7ca74640 1 -- 172.21.15.60:0/2219821576 shutdown_connections 2024-05-09T05:02:04.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:04.442+0000 7fbd7ca74640 1 -- 172.21.15.60:0/2219821576 wait complete. 2024-05-09T05:02:05.455 INFO:tasks.ceph:Scrubbing osd.0 2024-05-09T05:02:05.456 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-05-09T05:02:05.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.576+0000 7facb2124640 1 Processor -- start 2024-05-09T05:02:05.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.577+0000 7facb2124640 1 -- start start 2024-05-09T05:02:05.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.577+0000 7facb2124640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7facac146b50 0x7facac0b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:05.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.577+0000 7facb2124640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7facac15d7a0 0x7facac15dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:05.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.577+0000 7facb2124640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facac0d0d40 con 0x7facac0d1de0 2024-05-09T05:02:05.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.577+0000 7facb2124640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7facac0d1020 con 0x7facac146b50 2024-05-09T05:02:05.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.577+0000 7facb2124640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7facac0d1300 con 0x7facac15d7a0 2024-05-09T05:02:05.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.578+0000 7facab7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7facac146b50 0x7facac0b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:05.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.578+0000 7facab7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7facac146b50 0x7facac0b5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36362/0 (socket says 172.21.15.60:36362) 2024-05-09T05:02:05.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.578+0000 7facab7fe640 1 -- 172.21.15.60:0/356193935 learned_addr learned my addr 172.21.15.60:0/356193935 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:05.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.578+0000 7facaaffd640 1 --2- 172.21.15.60:0/356193935 >> v2:172.21.15.114:3300/0 conn(0x7facac15d7a0 0x7facac15dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:05.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.578+0000 7facab7fe640 1 -- 172.21.15.60:0/356193935 >> v2:172.21.15.114:3300/0 conn(0x7facac15d7a0 msgr2=0x7facac15dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:05.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.578+0000 7facab7fe640 1 --2- 172.21.15.60:0/356193935 >> v2:172.21.15.114:3300/0 conn(0x7facac15d7a0 0x7facac15dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:05.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.578+0000 7facab7fe640 1 -- 172.21.15.60:0/356193935 >> v1:172.21.15.114:6789/0 conn(0x7facac0d1de0 legacy=0x7facac0b6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:05.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.578+0000 7facab7fe640 1 -- 172.21.15.60:0/356193935 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facac0d15e0 con 0x7facac146b50 2024-05-09T05:02:05.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.579+0000 7facab7fe640 1 --2- 172.21.15.60:0/356193935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facac146b50 0x7facac0b5ec0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7facac0b7d60 tx=0x7fac9c02a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33660fc615ca7a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:05.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.579+0000 7facaa7fc640 1 -- 172.21.15.60:0/356193935 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9c037070 con 0x7facac146b50 2024-05-09T05:02:05.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.579+0000 7facaa7fc640 1 -- 172.21.15.60:0/356193935 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac9c032440 con 0x7facac146b50 2024-05-09T05:02:05.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.579+0000 7facaa7fc640 1 -- 172.21.15.60:0/356193935 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9c032810 con 0x7facac146b50 2024-05-09T05:02:05.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.579+0000 7facb2124640 1 -- 172.21.15.60:0/356193935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facac146b50 msgr2=0x7facac0b5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:05.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.579+0000 7facb2124640 1 --2- 172.21.15.60:0/356193935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facac146b50 0x7facac0b5ec0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7facac0b7d60 tx=0x7fac9c02a150 comp rx=0 tx=0).stop 2024-05-09T05:02:05.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.580+0000 7facb2124640 1 -- 172.21.15.60:0/356193935 shutdown_connections 2024-05-09T05:02:05.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.580+0000 7facb2124640 1 --2- 172.21.15.60:0/356193935 >> v2:172.21.15.114:3300/0 conn(0x7facac15d7a0 0x7facac15dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:05.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.580+0000 7facb2124640 1 --2- 172.21.15.60:0/356193935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facac146b50 0x7facac0b5ec0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:05.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.580+0000 7facb2124640 1 -- 172.21.15.60:0/356193935 >> 172.21.15.60:0/356193935 conn(0x7facac06e580 msgr2=0x7facac0709a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:05.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.580+0000 7facb2124640 1 -- 172.21.15.60:0/356193935 shutdown_connections 2024-05-09T05:02:05.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.580+0000 7facb2124640 1 -- 172.21.15.60:0/356193935 wait complete. 2024-05-09T05:02:05.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.581+0000 7facb2124640 1 Processor -- start 2024-05-09T05:02:05.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.581+0000 7facb2124640 1 -- start start 2024-05-09T05:02:05.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.582+0000 7facb2124640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facac0d1de0 0x7facac13e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:05.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.582+0000 7facb2124640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facac146b50 0x7facac06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:05.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.582+0000 7facb2124640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facac15d7a0 0x7facac06c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:05.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.582+0000 7facb2124640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7facac146f30 con 0x7facac146b50 2024-05-09T05:02:05.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.582+0000 7facb2124640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7facac0b6400 con 0x7facac0d1de0 2024-05-09T05:02:05.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.582+0000 7facb2124640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7facac1470b0 con 0x7facac15d7a0 2024-05-09T05:02:05.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.582+0000 7facaaffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facac146b50 0x7facac06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:05.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.582+0000 7facaaffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facac146b50 0x7facac06c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36378/0 (socket says 172.21.15.60:36378) 2024-05-09T05:02:05.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.582+0000 7facaaffd640 1 -- 172.21.15.60:0/1873636878 learned_addr learned my addr 172.21.15.60:0/1873636878 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:05.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.582+0000 7facab7fe640 1 --2- 172.21.15.60:0/1873636878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facac0d1de0 0x7facac13e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:05.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.582+0000 7facabfff640 1 --2- 172.21.15.60:0/1873636878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facac15d7a0 0x7facac06c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:05.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.582+0000 7facaaffd640 1 -- 172.21.15.60:0/1873636878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facac15d7a0 msgr2=0x7facac06c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:05.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.582+0000 7facaaffd640 1 --2- 172.21.15.60:0/1873636878 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7facac15d7a0 0x7facac06c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:05.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.582+0000 7facaaffd640 1 -- 172.21.15.60:0/1873636878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facac0d1de0 msgr2=0x7facac13e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:05.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.582+0000 7facaaffd640 1 --2- 172.21.15.60:0/1873636878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7facac0d1de0 0x7facac13e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:05.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.582+0000 7facaaffd640 1 -- 172.21.15.60:0/1873636878 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facac13b7b0 con 0x7facac146b50 2024-05-09T05:02:05.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.582+0000 7facaaffd640 1 --2- 172.21.15.60:0/1873636878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facac146b50 0x7facac06c370 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7faca000e870 tx=0x7faca000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:05.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.583+0000 7fac93fff640 1 -- 172.21.15.60:0/1873636878 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca0017070 con 0x7facac146b50 2024-05-09T05:02:05.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.583+0000 7fac93fff640 1 -- 172.21.15.60:0/1873636878 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faca000c940 con 0x7facac146b50 2024-05-09T05:02:05.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.583+0000 7facb2124640 1 -- 172.21.15.60:0/1873636878 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facac06d180 con 0x7facac146b50 2024-05-09T05:02:05.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.583+0000 7fac93fff640 1 -- 172.21.15.60:0/1873636878 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca000cd30 con 0x7facac146b50 2024-05-09T05:02:05.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.583+0000 7facb2124640 1 -- 172.21.15.60:0/1873636878 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facac06d740 con 0x7facac146b50 2024-05-09T05:02:05.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.584+0000 7facb2124640 1 -- 172.21.15.60:0/1873636878 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7facac051040 con 0x7facac146b50 2024-05-09T05:02:05.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.584+0000 7fac93fff640 1 -- 172.21.15.60:0/1873636878 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faca001a070 con 0x7facac146b50 2024-05-09T05:02:05.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.585+0000 7fac93fff640 1 --2- 172.21.15.60:0/1873636878 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fac98076300 0x7fac980787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:05.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.585+0000 7fac93fff640 1 -- 172.21.15.60:0/1873636878 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7faca0097650 con 0x7facac146b50 2024-05-09T05:02:05.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.585+0000 7facab7fe640 1 --2- 172.21.15.60:0/1873636878 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fac98076300 0x7fac980787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:05.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.585+0000 7fac93fff640 1 --2- 172.21.15.60:0/1873636878 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x7fac9807c7a0 0x7fac9807ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:05.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.585+0000 7fac93fff640 1 -- 172.21.15.60:0/1873636878 --> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faca0016ec0 con 0x7fac9807c7a0 2024-05-09T05:02:05.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.585+0000 7fac93fff640 1 -- 172.21.15.60:0/1873636878 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7faca0097ae0 con 0x7facac146b50 2024-05-09T05:02:05.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.585+0000 7facabfff640 1 --2- 172.21.15.60:0/1873636878 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x7fac9807c7a0 0x7fac9807ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:05.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.586+0000 7facabfff640 1 --2- 172.21.15.60:0/1873636878 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x7fac9807c7a0 0x7fac9807ebe0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:05.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.586+0000 7facab7fe640 1 --2- 172.21.15.60:0/1873636878 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fac98076300 0x7fac980787c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fac9c034000 tx=0x7fac9c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:05.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.586+0000 7fac93fff640 1 -- 172.21.15.60:0/1873636878 <== osd.0 v2:172.21.15.60:6800/1298086532 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faca0016ec0 con 0x7fac9807c7a0 2024-05-09T05:02:05.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.600+0000 7facb2124640 1 -- 172.21.15.60:0/1873636878 --> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7facac0b2c60 con 0x7fac9807c7a0 2024-05-09T05:02:05.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.600+0000 7fac93fff640 1 -- 172.21.15.60:0/1873636878 <== osd.0 v2:172.21.15.60:6800/1298086532 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7facac0b2c60 con 0x7fac9807c7a0 2024-05-09T05:02:05.601 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:02:05.602 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T05:02:05.602 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:02:05.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.600+0000 7facb2124640 1 -- 172.21.15.60:0/1873636878 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x7fac9807c7a0 msgr2=0x7fac9807ebe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:05.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.600+0000 7facb2124640 1 --2- 172.21.15.60:0/1873636878 >> [v2:172.21.15.60:6800/1298086532,v1:172.21.15.60:6801/1298086532] conn(0x7fac9807c7a0 0x7fac9807ebe0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:05.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.600+0000 7facb2124640 1 -- 172.21.15.60:0/1873636878 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fac98076300 msgr2=0x7fac980787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:05.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.600+0000 7facb2124640 1 --2- 172.21.15.60:0/1873636878 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fac98076300 0x7fac980787c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fac9c034000 tx=0x7fac9c02e000 comp rx=0 tx=0).stop 2024-05-09T05:02:05.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.600+0000 7facb2124640 1 -- 172.21.15.60:0/1873636878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facac146b50 msgr2=0x7facac06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:05.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.600+0000 7facb2124640 1 --2- 172.21.15.60:0/1873636878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facac146b50 0x7facac06c370 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7faca000e870 tx=0x7faca000ed30 comp rx=0 tx=0).stop 2024-05-09T05:02:05.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.600+0000 7facabfff640 1 -- 172.21.15.60:0/1873636878 reap_dead start 2024-05-09T05:02:05.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.600+0000 7facb2124640 1 -- 172.21.15.60:0/1873636878 shutdown_connections 2024-05-09T05:02:05.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.600+0000 7facb2124640 1 -- 172.21.15.60:0/1873636878 >> 172.21.15.60:0/1873636878 conn(0x7facac06e580 msgr2=0x7facac0b7580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:05.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.600+0000 7facb2124640 1 -- 172.21.15.60:0/1873636878 shutdown_connections 2024-05-09T05:02:05.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.600+0000 7facb2124640 1 -- 172.21.15.60:0/1873636878 wait complete. 2024-05-09T05:02:05.612 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-05-09T05:02:05.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.716+0000 7fe0e5be4640 1 Processor -- start 2024-05-09T05:02:05.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.716+0000 7fe0e5be4640 1 -- start start 2024-05-09T05:02:05.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.717+0000 7fe0e5be4640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe0e015f3c0 0x7fe0e009ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:05.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.717+0000 7fe0e5be4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0e00b4d30 0x7fe0e0072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.717+0000 7fe0e5be4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0e015f7a0 con 0x7fe0e0073fb0 2024-05-09T05:02:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.717+0000 7fe0e5be4640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fe0e0071cb0 con 0x7fe0e015f3c0 2024-05-09T05:02:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.717+0000 7fe0e5be4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe0e009f480 con 0x7fe0e00b4d30 2024-05-09T05:02:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.717+0000 7fe0dffff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0e00b4d30 0x7fe0e0072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.717+0000 7fe0df7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe0e015f3c0 0x7fe0e009ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.717+0000 7fe0df7fe640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fe0e015f3c0 0x7fe0e009ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.60:32866/0 (socket says 172.21.15.60:32866) 2024-05-09T05:02:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.717+0000 7fe0d6ffd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe0e0073fb0 0x7fe0e00b4620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.60:40402/0 (socket says 172.21.15.60:40402) 2024-05-09T05:02:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.717+0000 7fe0df7fe640 1 -- 172.21.15.60:0/1626980251 learned_addr learned my addr 172.21.15.60:0/1626980251 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.717+0000 7fe0dffff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0e00b4d30 0x7fe0e0072140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.60:60516/0 (socket says 172.21.15.60:60516) 2024-05-09T05:02:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.718+0000 7fe0df7fe640 1 -- 172.21.15.60:0/1626980251 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0e00b4d30 msgr2=0x7fe0e0072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.718+0000 7fe0df7fe640 1 --2- 172.21.15.60:0/1626980251 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0e00b4d30 0x7fe0e0072140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.718+0000 7fe0df7fe640 1 -- 172.21.15.60:0/1626980251 >> v1:172.21.15.114:6789/0 conn(0x7fe0e0073fb0 legacy=0x7fe0e00b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.718+0000 7fe0df7fe640 1 -- 172.21.15.60:0/1626980251 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e009f760 con 0x7fe0e015f3c0 2024-05-09T05:02:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.718+0000 7fe0deffd640 1 -- 172.21.15.60:0/1626980251 <== mon.1 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1015160309 0 0) 0x7fe0e015f7a0 con 0x7fe0e0073fb0 2024-05-09T05:02:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.718+0000 7fe0dffff640 1 --2- 172.21.15.60:0/1626980251 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0e00b4d30 0x7fe0e0072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:02:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.718+0000 7fe0df7fe640 1 --2- 172.21.15.60:0/1626980251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0e015f3c0 0x7fe0e009ebe0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0009870 tx=0x7fe0d002a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6662d5b2abedce5d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.719+0000 7fe0deffd640 1 -- 172.21.15.60:0/1626980251 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d0037070 con 0x7fe0e015f3c0 2024-05-09T05:02:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.719+0000 7fe0deffd640 1 -- 172.21.15.60:0/1626980251 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0d0032440 con 0x7fe0e015f3c0 2024-05-09T05:02:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.719+0000 7fe0deffd640 1 -- 172.21.15.60:0/1626980251 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d0032810 con 0x7fe0e015f3c0 2024-05-09T05:02:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.719+0000 7fe0e5be4640 1 -- 172.21.15.60:0/1626980251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0e015f3c0 msgr2=0x7fe0e009ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.719+0000 7fe0e5be4640 1 --2- 172.21.15.60:0/1626980251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0e015f3c0 0x7fe0e009ebe0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0009870 tx=0x7fe0d002a1c0 comp rx=0 tx=0).stop 2024-05-09T05:02:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.719+0000 7fe0e5be4640 1 -- 172.21.15.60:0/1626980251 shutdown_connections 2024-05-09T05:02:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.719+0000 7fe0e5be4640 1 --2- 172.21.15.60:0/1626980251 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0e00b4d30 0x7fe0e0072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.719+0000 7fe0e5be4640 1 --2- 172.21.15.60:0/1626980251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0e015f3c0 0x7fe0e009ebe0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.719+0000 7fe0e5be4640 1 -- 172.21.15.60:0/1626980251 >> 172.21.15.60:0/1626980251 conn(0x7fe0e00d1470 msgr2=0x7fe0e00d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.719+0000 7fe0e5be4640 1 -- 172.21.15.60:0/1626980251 shutdown_connections 2024-05-09T05:02:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.719+0000 7fe0e5be4640 1 -- 172.21.15.60:0/1626980251 wait complete. 2024-05-09T05:02:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.720+0000 7fe0e5be4640 1 Processor -- start 2024-05-09T05:02:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.721+0000 7fe0e5be4640 1 -- start start 2024-05-09T05:02:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.721+0000 7fe0e5be4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0e0073fb0 0x7fe0e01f7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.721+0000 7fe0e5be4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0e00b4d30 0x7fe0e01f5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.721+0000 7fe0e5be4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0e015f3c0 0x7fe0e009b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.721+0000 7fe0e5be4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e0072680 con 0x7fe0e00b4d30 2024-05-09T05:02:05.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.721+0000 7fe0e5be4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e009f120 con 0x7fe0e0073fb0 2024-05-09T05:02:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.721+0000 7fe0e5be4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fe0e0072800 con 0x7fe0e015f3c0 2024-05-09T05:02:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.722+0000 7fe0d6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0e00b4d30 0x7fe0e01f5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.722+0000 7fe0df7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0e0073fb0 0x7fe0e01f7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.722+0000 7fe0d6ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0e00b4d30 0x7fe0e01f5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36392/0 (socket says 172.21.15.60:36392) 2024-05-09T05:02:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.722+0000 7fe0df7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0e0073fb0 0x7fe0e01f7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.60:32870/0 (socket says 172.21.15.60:32870) 2024-05-09T05:02:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.722+0000 7fe0dffff640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0e015f3c0 0x7fe0e009b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.722+0000 7fe0d6ffd640 1 -- 172.21.15.60:0/2685643894 learned_addr learned my addr 172.21.15.60:0/2685643894 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:05.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.722+0000 7fe0dffff640 1 -- 172.21.15.60:0/2685643894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0e0073fb0 msgr2=0x7fe0e01f7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:05.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.722+0000 7fe0dffff640 1 --2- 172.21.15.60:0/2685643894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0e0073fb0 0x7fe0e01f7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:05.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.722+0000 7fe0dffff640 1 -- 172.21.15.60:0/2685643894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0e00b4d30 msgr2=0x7fe0e01f5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:05.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.722+0000 7fe0dffff640 1 --2- 172.21.15.60:0/2685643894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0e00b4d30 0x7fe0e01f5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:05.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.722+0000 7fe0dffff640 1 -- 172.21.15.60:0/2685643894 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e01f63a0 con 0x7fe0e015f3c0 2024-05-09T05:02:05.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.722+0000 7fe0dffff640 1 --2- 172.21.15.60:0/2685643894 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0e015f3c0 0x7fe0e009b530 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00d52f0 tx=0x7fe0d800ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:05.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.722+0000 7fe0dcff9640 1 -- 172.21.15.60:0/2685643894 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d8018020 con 0x7fe0e015f3c0 2024-05-09T05:02:05.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.722+0000 7fe0dcff9640 1 -- 172.21.15.60:0/2685643894 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0d8014480 con 0x7fe0e015f3c0 2024-05-09T05:02:05.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.722+0000 7fe0e5be4640 1 -- 172.21.15.60:0/2685643894 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0e00c8580 con 0x7fe0e015f3c0 2024-05-09T05:02:05.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.722+0000 7fe0dcff9640 1 -- 172.21.15.60:0/2685643894 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d8014870 con 0x7fe0e015f3c0 2024-05-09T05:02:05.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.722+0000 7fe0e5be4640 1 -- 172.21.15.60:0/2685643894 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0e00c8b40 con 0x7fe0e015f3c0 2024-05-09T05:02:05.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.723+0000 7fe0e5be4640 1 -- 172.21.15.60:0/2685643894 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0e0051040 con 0x7fe0e015f3c0 2024-05-09T05:02:05.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.724+0000 7fe0dcff9640 1 -- 172.21.15.60:0/2685643894 <== mon.2 v2:172.21.15.114:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe0d8005d10 con 0x7fe0e015f3c0 2024-05-09T05:02:05.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.724+0000 7fe0dcff9640 1 --2- 172.21.15.60:0/2685643894 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe0c0076300 0x7fe0c00787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:05.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.725+0000 7fe0dcff9640 1 -- 172.21.15.60:0/2685643894 <== mon.2 v2:172.21.15.114:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7fe0d8098150 con 0x7fe0e015f3c0 2024-05-09T05:02:05.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.725+0000 7fe0df7fe640 1 --2- 172.21.15.60:0/2685643894 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe0c0076300 0x7fe0c00787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:05.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.725+0000 7fe0df7fe640 1 --2- 172.21.15.60:0/2685643894 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe0c0076300 0x7fe0c00787c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0034000 tx=0x7fe0d002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:05.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.729+0000 7fe0dcff9640 1 -- 172.21.15.60:0/2685643894 <== mon.2 v2:172.21.15.114:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe0d8022030 con 0x7fe0e015f3c0 2024-05-09T05:02:05.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.925+0000 7fe0e5be4640 1 -- 172.21.15.60:0/2685643894 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fe0e00c8330 con 0x7fe0c0076300 2024-05-09T05:02:05.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.925+0000 7fe0dcff9640 1 -- 172.21.15.60:0/2685643894 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe0e00c8330 con 0x7fe0c0076300 2024-05-09T05:02:05.926 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 0 to deep-scrub 2024-05-09T05:02:05.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.928+0000 7fe0e5be4640 1 -- 172.21.15.60:0/2685643894 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe0c0076300 msgr2=0x7fe0c00787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:05.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.928+0000 7fe0e5be4640 1 --2- 172.21.15.60:0/2685643894 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe0c0076300 0x7fe0c00787c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0034000 tx=0x7fe0d002e000 comp rx=0 tx=0).stop 2024-05-09T05:02:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.928+0000 7fe0e5be4640 1 -- 172.21.15.60:0/2685643894 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0e015f3c0 msgr2=0x7fe0e009b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.928+0000 7fe0e5be4640 1 --2- 172.21.15.60:0/2685643894 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0e015f3c0 0x7fe0e009b530 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00d52f0 tx=0x7fe0d800ef90 comp rx=0 tx=0).stop 2024-05-09T05:02:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.928+0000 7fe0e5be4640 1 -- 172.21.15.60:0/2685643894 shutdown_connections 2024-05-09T05:02:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.928+0000 7fe0e5be4640 1 --2- 172.21.15.60:0/2685643894 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fe0c0076300 0x7fe0c00787c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.928+0000 7fe0e5be4640 1 --2- 172.21.15.60:0/2685643894 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fe0e015f3c0 0x7fe0e009b530 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.928+0000 7fe0e5be4640 1 --2- 172.21.15.60:0/2685643894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0e00b4d30 0x7fe0e01f5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.928+0000 7fe0e5be4640 1 --2- 172.21.15.60:0/2685643894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0e0073fb0 0x7fe0e01f7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.928+0000 7fe0e5be4640 1 -- 172.21.15.60:0/2685643894 >> 172.21.15.60:0/2685643894 conn(0x7fe0e00d1470 msgr2=0x7fe0e01619c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.928+0000 7fe0e5be4640 1 -- 172.21.15.60:0/2685643894 shutdown_connections 2024-05-09T05:02:05.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:05.928+0000 7fe0e5be4640 1 -- 172.21.15.60:0/2685643894 wait complete. 2024-05-09T05:02:05.939 INFO:tasks.ceph:Scrubbing osd.1 2024-05-09T05:02:05.939 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-05-09T05:02:06.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.047+0000 7fcccb459640 1 Processor -- start 2024-05-09T05:02:06.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.047+0000 7fcccb459640 1 -- start start 2024-05-09T05:02:06.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.048+0000 7fcccb459640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fccc41481b0 0x7fccc40b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.048+0000 7fcccb459640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccc4148710 0x7fccc40d45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.048+0000 7fcccb459640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccc40b5a90 con 0x7fccc40d4af0 2024-05-09T05:02:06.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.048+0000 7fcccb459640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fccc40b5d70 con 0x7fccc41481b0 2024-05-09T05:02:06.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.048+0000 7fcccb459640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fccc40b6050 con 0x7fccc4148710 2024-05-09T05:02:06.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.048+0000 7fccc91f5640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fccc41481b0 0x7fccc40b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.048+0000 7fccc89f4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccc4148710 0x7fccc40d45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.048+0000 7fccc91f5640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fccc41481b0 0x7fccc40b54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.60:32878/0 (socket says 172.21.15.60:32878) 2024-05-09T05:02:06.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.048+0000 7fccc99f6640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fccc40d4af0 0x7fccc4162ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.60:40414/0 (socket says 172.21.15.60:40414) 2024-05-09T05:02:06.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.048+0000 7fccc91f5640 1 -- 172.21.15.60:0/2691288879 learned_addr learned my addr 172.21.15.60:0/2691288879 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:06.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.048+0000 7fccc89f4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccc4148710 0x7fccc40d45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3301/0 says I am v2:172.21.15.60:60538/0 (socket says 172.21.15.60:60538) 2024-05-09T05:02:06.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.049+0000 7fccc91f5640 1 -- 172.21.15.60:0/2691288879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccc4148710 msgr2=0x7fccc40d45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:06.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.049+0000 7fccc91f5640 1 --2- 172.21.15.60:0/2691288879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccc4148710 0x7fccc40d45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.049+0000 7fccc91f5640 1 -- 172.21.15.60:0/2691288879 >> v1:172.21.15.114:6789/0 conn(0x7fccc40d4af0 legacy=0x7fccc4162ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.049+0000 7fccc91f5640 1 -- 172.21.15.60:0/2691288879 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccc40b6330 con 0x7fccc41481b0 2024-05-09T05:02:06.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.049+0000 7fccc89f4640 1 --2- 172.21.15.60:0/2691288879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccc4148710 0x7fccc40d45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:02:06.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.050+0000 7fccc91f5640 1 --2- 172.21.15.60:0/2691288879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc41481b0 0x7fccc40b54b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fccb4009e50 tx=0x7fccb402a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5d681d654751337 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:06.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.050+0000 7fccc3fff640 1 -- 172.21.15.60:0/2691288879 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb4036070 con 0x7fccc41481b0 2024-05-09T05:02:06.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.050+0000 7fccc3fff640 1 -- 172.21.15.60:0/2691288879 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccb4031600 con 0x7fccc41481b0 2024-05-09T05:02:06.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.050+0000 7fccc3fff640 1 -- 172.21.15.60:0/2691288879 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb40319d0 con 0x7fccc41481b0 2024-05-09T05:02:06.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.050+0000 7fcccb459640 1 -- 172.21.15.60:0/2691288879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc41481b0 msgr2=0x7fccc40b54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.050+0000 7fcccb459640 1 --2- 172.21.15.60:0/2691288879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc41481b0 0x7fccc40b54b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fccb4009e50 tx=0x7fccb402a680 comp rx=0 tx=0).stop 2024-05-09T05:02:06.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.050+0000 7fcccb459640 1 -- 172.21.15.60:0/2691288879 shutdown_connections 2024-05-09T05:02:06.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.050+0000 7fcccb459640 1 --2- 172.21.15.60:0/2691288879 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccc4148710 0x7fccc40d45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.050+0000 7fcccb459640 1 --2- 172.21.15.60:0/2691288879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc41481b0 0x7fccc40b54b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.050+0000 7fcccb459640 1 -- 172.21.15.60:0/2691288879 >> 172.21.15.60:0/2691288879 conn(0x7fccc406f640 msgr2=0x7fccc4073840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:06.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.050+0000 7fcccb459640 1 -- 172.21.15.60:0/2691288879 shutdown_connections 2024-05-09T05:02:06.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.050+0000 7fcccb459640 1 -- 172.21.15.60:0/2691288879 wait complete. 2024-05-09T05:02:06.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.052+0000 7fcccb459640 1 Processor -- start 2024-05-09T05:02:06.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.052+0000 7fcccb459640 1 -- start start 2024-05-09T05:02:06.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.052+0000 7fcccb459640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccc40d4af0 0x7fccc415e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.052+0000 7fcccb459640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc41481b0 0x7fccc40b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.052+0000 7fcccb459640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccc4148710 0x7fccc41427f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.052+0000 7fcccb459640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fccc40d2ea0 con 0x7fccc40d4af0 2024-05-09T05:02:06.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.052+0000 7fcccb459640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fccc40d2d20 con 0x7fccc41481b0 2024-05-09T05:02:06.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.052+0000 7fcccb459640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fccc41635e0 con 0x7fccc4148710 2024-05-09T05:02:06.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.053+0000 7fccc91f5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccc40d4af0 0x7fccc415e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.053+0000 7fccc91f5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccc40d4af0 0x7fccc415e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36394/0 (socket says 172.21.15.60:36394) 2024-05-09T05:02:06.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.053+0000 7fccc89f4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc41481b0 0x7fccc40b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.053+0000 7fccc99f6640 1 --2- 172.21.15.60:0/3035582183 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccc4148710 0x7fccc41427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.053+0000 7fccc91f5640 1 -- 172.21.15.60:0/3035582183 learned_addr learned my addr 172.21.15.60:0/3035582183 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:06.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.053+0000 7fccc91f5640 1 -- 172.21.15.60:0/3035582183 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccc4148710 msgr2=0x7fccc41427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.053+0000 7fccc91f5640 1 --2- 172.21.15.60:0/3035582183 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fccc4148710 0x7fccc41427f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.053+0000 7fccc91f5640 1 -- 172.21.15.60:0/3035582183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc41481b0 msgr2=0x7fccc40b40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.053+0000 7fccc91f5640 1 --2- 172.21.15.60:0/3035582183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccc41481b0 0x7fccc40b40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.053+0000 7fccc91f5640 1 -- 172.21.15.60:0/3035582183 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccc4160330 con 0x7fccc40d4af0 2024-05-09T05:02:06.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.053+0000 7fccc91f5640 1 --2- 172.21.15.60:0/3035582183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccc40d4af0 0x7fccc415e980 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fccb4034040 tx=0x7fccb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:06.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.053+0000 7fccc1ffb640 1 -- 172.21.15.60:0/3035582183 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb4036050 con 0x7fccc40d4af0 2024-05-09T05:02:06.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.053+0000 7fccc1ffb640 1 -- 172.21.15.60:0/3035582183 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccb4042070 con 0x7fccc40d4af0 2024-05-09T05:02:06.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.053+0000 7fcccb459640 1 -- 172.21.15.60:0/3035582183 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccc4160610 con 0x7fccc40d4af0 2024-05-09T05:02:06.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.053+0000 7fccc1ffb640 1 -- 172.21.15.60:0/3035582183 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb4009510 con 0x7fccc40d4af0 2024-05-09T05:02:06.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.053+0000 7fcccb459640 1 -- 172.21.15.60:0/3035582183 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccc4160bd0 con 0x7fccc40d4af0 2024-05-09T05:02:06.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.054+0000 7fcccb459640 1 -- 172.21.15.60:0/3035582183 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fccc4160fb0 con 0x7fccc40d4af0 2024-05-09T05:02:06.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.055+0000 7fccc1ffb640 1 -- 172.21.15.60:0/3035582183 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fccb4007c70 con 0x7fccc40d4af0 2024-05-09T05:02:06.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.055+0000 7fccc1ffb640 1 --2- 172.21.15.60:0/3035582183 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fccb00762d0 0x7fccb0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.056+0000 7fccc1ffb640 1 -- 172.21.15.60:0/3035582183 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7fccb40b8c50 con 0x7fccc40d4af0 2024-05-09T05:02:06.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.056+0000 7fccc1ffb640 1 --2- 172.21.15.60:0/3035582183 >> [v2:172.21.15.60:6802/291519275,v1:172.21.15.60:6804/291519275] conn(0x7fccb007c770 0x7fccb007ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.056+0000 7fccc1ffb640 1 -- 172.21.15.60:0/3035582183 --> [v2:172.21.15.60:6802/291519275,v1:172.21.15.60:6804/291519275] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fccb007f280 con 0x7fccb007c770 2024-05-09T05:02:06.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.056+0000 7fccc1ffb640 1 -- 172.21.15.60:0/3035582183 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fccb40b9000 con 0x7fccc40d4af0 2024-05-09T05:02:06.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.056+0000 7fccc89f4640 1 --2- 172.21.15.60:0/3035582183 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fccb00762d0 0x7fccb0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.056+0000 7fccc99f6640 1 --2- 172.21.15.60:0/3035582183 >> [v2:172.21.15.60:6802/291519275,v1:172.21.15.60:6804/291519275] conn(0x7fccb007c770 0x7fccb007ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.056+0000 7fccc99f6640 1 --2- 172.21.15.60:0/3035582183 >> [v2:172.21.15.60:6802/291519275,v1:172.21.15.60:6804/291519275] conn(0x7fccb007c770 0x7fccb007ebb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:06.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.056+0000 7fccc89f4640 1 --2- 172.21.15.60:0/3035582183 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fccb00762d0 0x7fccb0078790 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009560 tx=0x7fccb8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:06.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.057+0000 7fccc1ffb640 1 -- 172.21.15.60:0/3035582183 <== osd.1 v2:172.21.15.60:6802/291519275 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fccb007f280 con 0x7fccb007c770 2024-05-09T05:02:06.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.070+0000 7fcccb459640 1 -- 172.21.15.60:0/3035582183 --> [v2:172.21.15.60:6802/291519275,v1:172.21.15.60:6804/291519275] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fccc4051040 con 0x7fccb007c770 2024-05-09T05:02:06.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.070+0000 7fccc1ffb640 1 -- 172.21.15.60:0/3035582183 <== osd.1 v2:172.21.15.60:6802/291519275 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fccc4051040 con 0x7fccb007c770 2024-05-09T05:02:06.071 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:02:06.072 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T05:02:06.072 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:02:06.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.071+0000 7fcccb459640 1 -- 172.21.15.60:0/3035582183 >> [v2:172.21.15.60:6802/291519275,v1:172.21.15.60:6804/291519275] conn(0x7fccb007c770 msgr2=0x7fccb007ebb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.071+0000 7fcccb459640 1 --2- 172.21.15.60:0/3035582183 >> [v2:172.21.15.60:6802/291519275,v1:172.21.15.60:6804/291519275] conn(0x7fccb007c770 0x7fccb007ebb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.071+0000 7fcccb459640 1 -- 172.21.15.60:0/3035582183 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fccb00762d0 msgr2=0x7fccb0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.071+0000 7fcccb459640 1 --2- 172.21.15.60:0/3035582183 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fccb00762d0 0x7fccb0078790 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009560 tx=0x7fccb8009190 comp rx=0 tx=0).stop 2024-05-09T05:02:06.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.071+0000 7fcccb459640 1 -- 172.21.15.60:0/3035582183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccc40d4af0 msgr2=0x7fccc415e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.071+0000 7fcccb459640 1 --2- 172.21.15.60:0/3035582183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccc40d4af0 0x7fccc415e980 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fccb4034040 tx=0x7fccb4004a40 comp rx=0 tx=0).stop 2024-05-09T05:02:06.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.071+0000 7fccc99f6640 1 -- 172.21.15.60:0/3035582183 reap_dead start 2024-05-09T05:02:06.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.071+0000 7fcccb459640 1 -- 172.21.15.60:0/3035582183 shutdown_connections 2024-05-09T05:02:06.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.071+0000 7fcccb459640 1 -- 172.21.15.60:0/3035582183 >> 172.21.15.60:0/3035582183 conn(0x7fccc406f640 msgr2=0x7fccc406fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:06.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.071+0000 7fcccb459640 1 -- 172.21.15.60:0/3035582183 shutdown_connections 2024-05-09T05:02:06.074 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.071+0000 7fcccb459640 1 -- 172.21.15.60:0/3035582183 wait complete. 2024-05-09T05:02:06.082 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-05-09T05:02:06.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.186+0000 7f9b553af640 1 Processor -- start 2024-05-09T05:02:06.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.186+0000 7f9b553af640 1 -- start start 2024-05-09T05:02:06.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.187+0000 7f9b553af640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b501598f0 0x7f9b50160ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.187+0000 7f9b553af640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f9b50072130 0x7f9b50072570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.187+0000 7f9b553af640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b500b2500 con 0x7f9b501605d0 2024-05-09T05:02:06.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.187+0000 7f9b553af640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f9b500addd0 con 0x7f9b50072130 2024-05-09T05:02:06.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.187+0000 7f9b553af640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9b50072ab0 con 0x7f9b501598f0 2024-05-09T05:02:06.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.187+0000 7f9b4e7fc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9b501605d0 0x7f9b500b1cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34254/0 (socket says 172.21.15.60:34254) 2024-05-09T05:02:06.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.187+0000 7f9b4e7fc640 1 -- 172.21.15.60:0/3689230595 learned_addr learned my addr 172.21.15.60:0/3689230595 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:02:06.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.188+0000 7f9b4effd640 1 --2- 172.21.15.60:0/3689230595 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b501598f0 0x7f9b50160ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.188+0000 7f9b4f7fe640 1 --2- 172.21.15.60:0/3689230595 >> v2:172.21.15.114:3300/0 conn(0x7f9b50072130 0x7f9b50072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.188+0000 7f9b4dffb640 1 -- 172.21.15.60:0/3689230595 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1189085080 0 0) 0x7f9b500b2500 con 0x7f9b501605d0 2024-05-09T05:02:06.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.188+0000 7f9b4dffb640 1 -- 172.21.15.60:0/3689230595 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b3c003520 con 0x7f9b501605d0 2024-05-09T05:02:06.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.188+0000 7f9b4effd640 1 -- 172.21.15.60:0/3689230595 >> v2:172.21.15.114:3300/0 conn(0x7f9b50072130 msgr2=0x7f9b50072570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:06.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.188+0000 7f9b4effd640 1 --2- 172.21.15.60:0/3689230595 >> v2:172.21.15.114:3300/0 conn(0x7f9b50072130 0x7f9b50072570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.188+0000 7f9b4effd640 1 -- 172.21.15.60:0/3689230595 >> v1:172.21.15.60:6789/0 conn(0x7f9b501605d0 legacy=0x7f9b500b1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.188+0000 7f9b4effd640 1 -- 172.21.15.60:0/3689230595 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b500b27e0 con 0x7f9b501598f0 2024-05-09T05:02:06.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.189+0000 7f9b4effd640 1 --2- 172.21.15.60:0/3689230595 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b501598f0 0x7f9b50160ae0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9b40009d90 tx=0x7f9b4002a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ee420f9cad229aa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:06.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.189+0000 7f9b4dffb640 1 -- 172.21.15.60:0/3689230595 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b40007740 con 0x7f9b501598f0 2024-05-09T05:02:06.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.189+0000 7f9b4dffb640 1 -- 172.21.15.60:0/3689230595 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b400078e0 con 0x7f9b501598f0 2024-05-09T05:02:06.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.189+0000 7f9b4dffb640 1 -- 172.21.15.60:0/3689230595 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b40007c90 con 0x7f9b501598f0 2024-05-09T05:02:06.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.189+0000 7f9b553af640 1 -- 172.21.15.60:0/3689230595 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b501598f0 msgr2=0x7f9b50160ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.189+0000 7f9b553af640 1 --2- 172.21.15.60:0/3689230595 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b501598f0 0x7f9b50160ae0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9b40009d90 tx=0x7f9b4002a530 comp rx=0 tx=0).stop 2024-05-09T05:02:06.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.189+0000 7f9b553af640 1 -- 172.21.15.60:0/3689230595 shutdown_connections 2024-05-09T05:02:06.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.189+0000 7f9b553af640 1 --2- 172.21.15.60:0/3689230595 >> v2:172.21.15.114:3300/0 conn(0x7f9b50072130 0x7f9b50072570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.189+0000 7f9b553af640 1 --2- 172.21.15.60:0/3689230595 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b501598f0 0x7f9b50160ae0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.189+0000 7f9b553af640 1 -- 172.21.15.60:0/3689230595 >> 172.21.15.60:0/3689230595 conn(0x7f9b500b4850 msgr2=0x7f9b500b4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:06.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.189+0000 7f9b553af640 1 -- 172.21.15.60:0/3689230595 shutdown_connections 2024-05-09T05:02:06.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.189+0000 7f9b553af640 1 -- 172.21.15.60:0/3689230595 wait complete. 2024-05-09T05:02:06.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.191+0000 7f9b553af640 1 Processor -- start 2024-05-09T05:02:06.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.191+0000 7f9b553af640 1 -- start start 2024-05-09T05:02:06.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.191+0000 7f9b553af640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b50072130 0x7f9b5015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.191+0000 7f9b553af640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b501598f0 0x7f9b5006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.191+0000 7f9b553af640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b501605d0 0x7f9b5006d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.191+0000 7f9b553af640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f9b50161020 con 0x7f9b50072130 2024-05-09T05:02:06.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.191+0000 7f9b553af640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9b500ae020 con 0x7f9b501605d0 2024-05-09T05:02:06.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.191+0000 7f9b553af640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f9b50159cd0 con 0x7f9b501598f0 2024-05-09T05:02:06.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.192+0000 7f9b4effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b50072130 0x7f9b5015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.192+0000 7f9b4effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b50072130 0x7f9b5015ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36400/0 (socket says 172.21.15.60:36400) 2024-05-09T05:02:06.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.192+0000 7f9b4effd640 1 -- 172.21.15.60:0/257554532 learned_addr learned my addr 172.21.15.60:0/257554532 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:06.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.192+0000 7f9b4e7fc640 1 --2- 172.21.15.60:0/257554532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b501598f0 0x7f9b5006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.192+0000 7f9b4f7fe640 1 --2- 172.21.15.60:0/257554532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b501605d0 0x7f9b5006d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.192+0000 7f9b4effd640 1 -- 172.21.15.60:0/257554532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b501598f0 msgr2=0x7f9b5006d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.192+0000 7f9b4effd640 1 --2- 172.21.15.60:0/257554532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b501598f0 0x7f9b5006d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.192+0000 7f9b4effd640 1 -- 172.21.15.60:0/257554532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b501605d0 msgr2=0x7f9b5006d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.192+0000 7f9b4effd640 1 --2- 172.21.15.60:0/257554532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b501605d0 0x7f9b5006d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.192+0000 7f9b4effd640 1 -- 172.21.15.60:0/257554532 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b501601a0 con 0x7f9b50072130 2024-05-09T05:02:06.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.192+0000 7f9b4effd640 1 --2- 172.21.15.60:0/257554532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b50072130 0x7f9b5015ae40 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9b40009d90 tx=0x7f9b40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:06.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.192+0000 7f9b477fe640 1 -- 172.21.15.60:0/257554532 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4003f070 con 0x7f9b50072130 2024-05-09T05:02:06.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.192+0000 7f9b477fe640 1 -- 172.21.15.60:0/257554532 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b40031c40 con 0x7f9b50072130 2024-05-09T05:02:06.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.192+0000 7f9b553af640 1 -- 172.21.15.60:0/257554532 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b5015a080 con 0x7f9b50072130 2024-05-09T05:02:06.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.192+0000 7f9b477fe640 1 -- 172.21.15.60:0/257554532 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b40009450 con 0x7f9b50072130 2024-05-09T05:02:06.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.192+0000 7f9b553af640 1 -- 172.21.15.60:0/257554532 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b5015a640 con 0x7f9b50072130 2024-05-09T05:02:06.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.193+0000 7f9b553af640 1 -- 172.21.15.60:0/257554532 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b50159150 con 0x7f9b50072130 2024-05-09T05:02:06.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.194+0000 7f9b477fe640 1 -- 172.21.15.60:0/257554532 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9b40003990 con 0x7f9b50072130 2024-05-09T05:02:06.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.194+0000 7f9b477fe640 1 --2- 172.21.15.60:0/257554532 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9b300762d0 0x7f9b30078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.195+0000 7f9b477fe640 1 -- 172.21.15.60:0/257554532 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7f9b400b8410 con 0x7f9b50072130 2024-05-09T05:02:06.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.195+0000 7f9b4e7fc640 1 --2- 172.21.15.60:0/257554532 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9b300762d0 0x7f9b30078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.195+0000 7f9b4e7fc640 1 --2- 172.21.15.60:0/257554532 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9b300762d0 0x7f9b30078790 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9b50161630 tx=0x7f9b38006aa0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:06.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.199+0000 7f9b477fe640 1 -- 172.21.15.60:0/257554532 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9b50159150 con 0x7f9b50072130 2024-05-09T05:02:06.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.393+0000 7f9b553af640 1 -- 172.21.15.60:0/257554532 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f9b50051040 con 0x7f9b300762d0 2024-05-09T05:02:06.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.393+0000 7f9b477fe640 1 -- 172.21.15.60:0/257554532 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9b50051040 con 0x7f9b300762d0 2024-05-09T05:02:06.394 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 1 to deep-scrub 2024-05-09T05:02:06.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.395+0000 7f9b553af640 1 -- 172.21.15.60:0/257554532 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9b300762d0 msgr2=0x7f9b30078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.395+0000 7f9b553af640 1 --2- 172.21.15.60:0/257554532 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9b300762d0 0x7f9b30078790 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9b50161630 tx=0x7f9b38006aa0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.395+0000 7f9b553af640 1 -- 172.21.15.60:0/257554532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b50072130 msgr2=0x7f9b5015ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.395+0000 7f9b553af640 1 --2- 172.21.15.60:0/257554532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b50072130 0x7f9b5015ae40 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9b40009d90 tx=0x7f9b40004a40 comp rx=0 tx=0).stop 2024-05-09T05:02:06.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.395+0000 7f9b553af640 1 -- 172.21.15.60:0/257554532 shutdown_connections 2024-05-09T05:02:06.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.395+0000 7f9b553af640 1 --2- 172.21.15.60:0/257554532 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f9b300762d0 0x7f9b30078790 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.395+0000 7f9b553af640 1 --2- 172.21.15.60:0/257554532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b501605d0 0x7f9b5006d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.395+0000 7f9b553af640 1 --2- 172.21.15.60:0/257554532 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f9b501598f0 0x7f9b5006d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.395+0000 7f9b553af640 1 --2- 172.21.15.60:0/257554532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b50072130 0x7f9b5015ae40 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.396+0000 7f9b553af640 1 -- 172.21.15.60:0/257554532 >> 172.21.15.60:0/257554532 conn(0x7f9b500b4850 msgr2=0x7f9b500b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:06.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.396+0000 7f9b553af640 1 -- 172.21.15.60:0/257554532 shutdown_connections 2024-05-09T05:02:06.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.396+0000 7f9b553af640 1 -- 172.21.15.60:0/257554532 wait complete. 2024-05-09T05:02:06.407 INFO:tasks.ceph:Scrubbing osd.2 2024-05-09T05:02:06.407 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-05-09T05:02:06.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.514+0000 7f458d578640 1 Processor -- start 2024-05-09T05:02:06.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.514+0000 7f458d578640 1 -- start start 2024-05-09T05:02:06.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.515+0000 7f458d578640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f45880b4ce0 0x7f458815e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.515+0000 7f458d578640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45880d17a0 con 0x7f4588072840 2024-05-09T05:02:06.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.515+0000 7f458d578640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45880d1a80 con 0x7f45880d0d50 2024-05-09T05:02:06.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.515+0000 7f458d578640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f458815db00 con 0x7f45880b4ce0 2024-05-09T05:02:06.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.515+0000 7f45877fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4588072840 0x7f4588160da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34256/0 (socket says 172.21.15.60:34256) 2024-05-09T05:02:06.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.515+0000 7f45877fe640 1 -- 172.21.15.60:0/955590536 learned_addr learned my addr 172.21.15.60:0/955590536 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:02:06.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.515+0000 7f4586ffd640 1 --2- 172.21.15.60:0/955590536 >> v2:172.21.15.114:3300/0 conn(0x7f45880b4ce0 0x7f458815e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.516+0000 7f4585ffb640 1 -- 172.21.15.60:0/955590536 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3534427664 0 0) 0x7f45880d17a0 con 0x7f4588072840 2024-05-09T05:02:06.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.516+0000 7f4585ffb640 1 -- 172.21.15.60:0/955590536 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4574003520 con 0x7f4588072840 2024-05-09T05:02:06.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.516+0000 7f4586ffd640 1 -- 172.21.15.60:0/955590536 >> v1:172.21.15.114:6789/0 conn(0x7f45880d0d50 legacy=0x7f4588072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.516+0000 7f4586ffd640 1 -- 172.21.15.60:0/955590536 >> v1:172.21.15.60:6789/0 conn(0x7f4588072840 legacy=0x7f4588160da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.516+0000 7f4586ffd640 1 -- 172.21.15.60:0/955590536 --> v2:172.21.15.114:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f458815d7b0 con 0x7f45880b4ce0 2024-05-09T05:02:06.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.517+0000 7f4586ffd640 1 --2- 172.21.15.60:0/955590536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45880b4ce0 0x7f458815e300 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f45800097a0 tx=0x7f458002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8025dbc4ea69634a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:06.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.517+0000 7f4585ffb640 1 -- 172.21.15.60:0/955590536 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4580037070 con 0x7f45880b4ce0 2024-05-09T05:02:06.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.517+0000 7f4585ffb640 1 -- 172.21.15.60:0/955590536 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4580032440 con 0x7f45880b4ce0 2024-05-09T05:02:06.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.517+0000 7f4585ffb640 1 -- 172.21.15.60:0/955590536 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45800327f0 con 0x7f45880b4ce0 2024-05-09T05:02:06.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.517+0000 7f458d578640 1 -- 172.21.15.60:0/955590536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45880b4ce0 msgr2=0x7f458815e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.517+0000 7f458d578640 1 --2- 172.21.15.60:0/955590536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45880b4ce0 0x7f458815e300 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f45800097a0 tx=0x7f458002a1a0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.517+0000 7f458d578640 1 -- 172.21.15.60:0/955590536 shutdown_connections 2024-05-09T05:02:06.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.517+0000 7f458d578640 1 --2- 172.21.15.60:0/955590536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45880b4ce0 0x7f458815e300 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.517+0000 7f458d578640 1 -- 172.21.15.60:0/955590536 >> 172.21.15.60:0/955590536 conn(0x7f458806f870 msgr2=0x7f458806fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:06.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.517+0000 7f458d578640 1 -- 172.21.15.60:0/955590536 shutdown_connections 2024-05-09T05:02:06.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.517+0000 7f458d578640 1 -- 172.21.15.60:0/955590536 wait complete. 2024-05-09T05:02:06.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.519+0000 7f458d578640 1 Processor -- start 2024-05-09T05:02:06.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.519+0000 7f458d578640 1 -- start start 2024-05-09T05:02:06.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.519+0000 7f458d578640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4588072840 0x7f4588153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.525 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.519+0000 7f458d578640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45880b4ce0 0x7f458815c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f458d578640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45880d0d50 0x7f458815c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f458d578640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f45880c74a0 con 0x7f45880d0d50 2024-05-09T05:02:06.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f458d578640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f45880c7320 con 0x7f45880b4ce0 2024-05-09T05:02:06.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f458d578640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f45880d1200 con 0x7f4588072840 2024-05-09T05:02:06.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f45877fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45880d0d50 0x7f458815c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f45877fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45880d0d50 0x7f458815c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36416/0 (socket says 172.21.15.60:36416) 2024-05-09T05:02:06.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f45877fe640 1 -- 172.21.15.60:0/86665149 learned_addr learned my addr 172.21.15.60:0/86665149 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:06.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f4586ffd640 1 --2- 172.21.15.60:0/86665149 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4588072840 0x7f4588153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f45867fc640 1 --2- 172.21.15.60:0/86665149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45880b4ce0 0x7f458815c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f45877fe640 1 -- 172.21.15.60:0/86665149 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4588072840 msgr2=0x7f4588153f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f45877fe640 1 --2- 172.21.15.60:0/86665149 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f4588072840 0x7f4588153f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f45877fe640 1 -- 172.21.15.60:0/86665149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45880b4ce0 msgr2=0x7f458815c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f45877fe640 1 --2- 172.21.15.60:0/86665149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45880b4ce0 0x7f458815c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f45877fe640 1 -- 172.21.15.60:0/86665149 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45881527f0 con 0x7f45880d0d50 2024-05-09T05:02:06.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f45867fc640 1 --2- 172.21.15.60:0/86665149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45880b4ce0 0x7f458815c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:02:06.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f45877fe640 1 --2- 172.21.15.60:0/86665149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45880d0d50 0x7f458815c750 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f457800cb40 tx=0x7f457800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:06.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f457f7fe640 1 -- 172.21.15.60:0/86665149 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457800aca0 con 0x7f45880d0d50 2024-05-09T05:02:06.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f457f7fe640 1 -- 172.21.15.60:0/86665149 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f457800ae40 con 0x7f45880d0d50 2024-05-09T05:02:06.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f458d578640 1 -- 172.21.15.60:0/86665149 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4588152ad0 con 0x7f45880d0d50 2024-05-09T05:02:06.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f457f7fe640 1 -- 172.21.15.60:0/86665149 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4578012600 con 0x7f45880d0d50 2024-05-09T05:02:06.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.520+0000 7f458d578640 1 -- 172.21.15.60:0/86665149 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4588153090 con 0x7f45880d0d50 2024-05-09T05:02:06.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.521+0000 7f458d578640 1 -- 172.21.15.60:0/86665149 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4588051040 con 0x7f45880d0d50 2024-05-09T05:02:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.522+0000 7f457f7fe640 1 -- 172.21.15.60:0/86665149 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f45780127a0 con 0x7f45880d0d50 2024-05-09T05:02:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.523+0000 7f457f7fe640 1 --2- 172.21.15.60:0/86665149 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f45680762d0 0x7f4568078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.523+0000 7f457f7fe640 1 -- 172.21.15.60:0/86665149 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7f4578097ab0 con 0x7f45880d0d50 2024-05-09T05:02:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.523+0000 7f457f7fe640 1 --2- 172.21.15.60:0/86665149 >> [v2:172.21.15.60:6816/483347218,v1:172.21.15.60:6817/483347218] conn(0x7f456807c770 0x7f456807ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.523+0000 7f457f7fe640 1 -- 172.21.15.60:0/86665149 --> [v2:172.21.15.60:6816/483347218,v1:172.21.15.60:6817/483347218] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f456807f280 con 0x7f456807c770 2024-05-09T05:02:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.523+0000 7f457f7fe640 1 -- 172.21.15.60:0/86665149 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f457809c2c0 con 0x7f45880d0d50 2024-05-09T05:02:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.523+0000 7f45867fc640 1 --2- 172.21.15.60:0/86665149 >> [v2:172.21.15.60:6816/483347218,v1:172.21.15.60:6817/483347218] conn(0x7f456807c770 0x7f456807ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.523+0000 7f4586ffd640 1 --2- 172.21.15.60:0/86665149 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f45680762d0 0x7f4568078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.523+0000 7f45867fc640 1 --2- 172.21.15.60:0/86665149 >> [v2:172.21.15.60:6816/483347218,v1:172.21.15.60:6817/483347218] conn(0x7f456807c770 0x7f456807ebb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:06.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.523+0000 7f4586ffd640 1 --2- 172.21.15.60:0/86665149 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f45680762d0 0x7f4568078790 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4580034000 tx=0x7f458002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:06.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.524+0000 7f457f7fe640 1 -- 172.21.15.60:0/86665149 <== osd.2 v2:172.21.15.60:6816/483347218 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f456807f280 con 0x7f456807c770 2024-05-09T05:02:06.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.537+0000 7f458d578640 1 -- 172.21.15.60:0/86665149 --> [v2:172.21.15.60:6816/483347218,v1:172.21.15.60:6817/483347218] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f45880709e0 con 0x7f456807c770 2024-05-09T05:02:06.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.537+0000 7f457f7fe640 1 -- 172.21.15.60:0/86665149 <== osd.2 v2:172.21.15.60:6816/483347218 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f45880709e0 con 0x7f456807c770 2024-05-09T05:02:06.539 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:02:06.539 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T05:02:06.540 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:02:06.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.538+0000 7f458d578640 1 -- 172.21.15.60:0/86665149 >> [v2:172.21.15.60:6816/483347218,v1:172.21.15.60:6817/483347218] conn(0x7f456807c770 msgr2=0x7f456807ebb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.538+0000 7f458d578640 1 --2- 172.21.15.60:0/86665149 >> [v2:172.21.15.60:6816/483347218,v1:172.21.15.60:6817/483347218] conn(0x7f456807c770 0x7f456807ebb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.538+0000 7f458d578640 1 -- 172.21.15.60:0/86665149 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f45680762d0 msgr2=0x7f4568078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.538+0000 7f458d578640 1 --2- 172.21.15.60:0/86665149 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f45680762d0 0x7f4568078790 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4580034000 tx=0x7f458002e000 comp rx=0 tx=0).stop 2024-05-09T05:02:06.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.538+0000 7f458d578640 1 -- 172.21.15.60:0/86665149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45880d0d50 msgr2=0x7f458815c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.538+0000 7f458d578640 1 --2- 172.21.15.60:0/86665149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45880d0d50 0x7f458815c750 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f457800cb40 tx=0x7f457800a060 comp rx=0 tx=0).stop 2024-05-09T05:02:06.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.538+0000 7f45877fe640 1 -- 172.21.15.60:0/86665149 reap_dead start 2024-05-09T05:02:06.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.538+0000 7f458d578640 1 -- 172.21.15.60:0/86665149 shutdown_connections 2024-05-09T05:02:06.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.538+0000 7f458d578640 1 -- 172.21.15.60:0/86665149 >> 172.21.15.60:0/86665149 conn(0x7f458806f870 msgr2=0x7f45880b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:06.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.538+0000 7f458d578640 1 -- 172.21.15.60:0/86665149 shutdown_connections 2024-05-09T05:02:06.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.538+0000 7f458d578640 1 -- 172.21.15.60:0/86665149 wait complete. 2024-05-09T05:02:06.549 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-05-09T05:02:06.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.654+0000 7f2f7af14640 1 Processor -- start 2024-05-09T05:02:06.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.654+0000 7f2f7af14640 1 -- start start 2024-05-09T05:02:06.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f7af14640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f741606f0 0x7f2f740d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f7af14640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f7406ff60 con 0x7f2f7415f300 2024-05-09T05:02:06.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f7af14640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f74070240 con 0x7f2f7406f9f0 2024-05-09T05:02:06.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f7af14640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f2f74070520 con 0x7f2f741606f0 2024-05-09T05:02:06.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f78cb0640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2f7415f300 0x7f2f741639f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34272/0 (socket says 172.21.15.60:34272) 2024-05-09T05:02:06.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f78cb0640 1 -- 172.21.15.60:0/2776950437 learned_addr learned my addr 172.21.15.60:0/2776950437 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:02:06.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f6bfff640 1 --2- 172.21.15.60:0/2776950437 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f741606f0 0x7f2f740d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f6b7fe640 1 -- 172.21.15.60:0/2776950437 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4155497111 0 0) 0x7f2f7406ff60 con 0x7f2f7415f300 2024-05-09T05:02:06.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f6b7fe640 1 -- 172.21.15.60:0/2776950437 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f5c003520 con 0x7f2f7415f300 2024-05-09T05:02:06.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f6b7fe640 1 -- 172.21.15.60:0/2776950437 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 422755010 0 0) 0x7f2f5c003520 con 0x7f2f7415f300 2024-05-09T05:02:06.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f6b7fe640 1 -- 172.21.15.60:0/2776950437 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f741606f0 msgr2=0x7f2f740d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:06.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f6b7fe640 1 --2- 172.21.15.60:0/2776950437 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f741606f0 0x7f2f740d33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f6b7fe640 1 -- 172.21.15.60:0/2776950437 >> v1:172.21.15.114:6789/0 conn(0x7f2f7406f9f0 legacy=0x7f2f7406ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f6b7fe640 1 -- 172.21.15.60:0/2776950437 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7406e740 con 0x7f2f7415f300 2024-05-09T05:02:06.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f6b7fe640 1 -- 172.21.15.60:0/2776950437 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f2f700016f0 con 0x7f2f7415f300 2024-05-09T05:02:06.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f6b7fe640 1 -- 172.21.15.60:0/2776950437 >> v1:172.21.15.60:6789/0 conn(0x7f2f7415f300 legacy=0x7f2f741639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f6b7fe640 1 --2- 172.21.15.60:0/2776950437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f5c003a30 0x7f2f5c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f6b7fe640 1 --2- 172.21.15.60:0/2776950437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f5c004800 0x7f2f5c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f6b7fe640 1 --2- 172.21.15.60:0/2776950437 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f741606f0 0x7f2f5c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f6b7fe640 1 -- 172.21.15.60:0/2776950437 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2f74070520 con 0x7f2f5c004800 2024-05-09T05:02:06.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f6b7fe640 1 -- 172.21.15.60:0/2776950437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5c00bb20 con 0x7f2f5c003a30 2024-05-09T05:02:06.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.655+0000 7f2f6b7fe640 1 -- 172.21.15.60:0/2776950437 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f2f5c00bd20 con 0x7f2f741606f0 2024-05-09T05:02:06.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.656+0000 7f2f6bfff640 1 --2- 172.21.15.60:0/2776950437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f5c004800 0x7f2f5c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.656+0000 7f2f78cb0640 1 --2- 172.21.15.60:0/2776950437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f5c003a30 0x7f2f5c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.656+0000 7f2f794b1640 1 --2- 172.21.15.60:0/2776950437 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f741606f0 0x7f2f5c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.656+0000 7f2f6bfff640 1 -- 172.21.15.60:0/2776950437 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f741606f0 msgr2=0x7f2f5c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:06.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.656+0000 7f2f6bfff640 1 --2- 172.21.15.60:0/2776950437 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f741606f0 0x7f2f5c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.656+0000 7f2f6bfff640 1 -- 172.21.15.60:0/2776950437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f5c003a30 msgr2=0x7f2f5c003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:06.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.656+0000 7f2f6bfff640 1 --2- 172.21.15.60:0/2776950437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f5c003a30 0x7f2f5c003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.656+0000 7f2f6bfff640 1 -- 172.21.15.60:0/2776950437 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f5c003520 con 0x7f2f5c004800 2024-05-09T05:02:06.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.657+0000 7f2f6bfff640 1 --2- 172.21.15.60:0/2776950437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f5c004800 0x7f2f5c004c80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2f64009ac0 tx=0x7f2f5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb66f6ae5c026df4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:06.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.657+0000 7f2f6b7fe640 1 -- 172.21.15.60:0/2776950437 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f64036070 con 0x7f2f5c004800 2024-05-09T05:02:06.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.657+0000 7f2f6b7fe640 1 -- 172.21.15.60:0/2776950437 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f64031400 con 0x7f2f5c004800 2024-05-09T05:02:06.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.657+0000 7f2f6b7fe640 1 -- 172.21.15.60:0/2776950437 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f64009690 con 0x7f2f5c004800 2024-05-09T05:02:06.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.657+0000 7f2f7af14640 1 -- 172.21.15.60:0/2776950437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f5c004800 msgr2=0x7f2f5c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.657+0000 7f2f7af14640 1 --2- 172.21.15.60:0/2776950437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f5c004800 0x7f2f5c004c80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2f64009ac0 tx=0x7f2f5c005830 comp rx=0 tx=0).stop 2024-05-09T05:02:06.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.657+0000 7f2f794b1640 1 -- 172.21.15.60:0/2776950437 reap_dead start 2024-05-09T05:02:06.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.657+0000 7f2f7af14640 1 -- 172.21.15.60:0/2776950437 shutdown_connections 2024-05-09T05:02:06.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.657+0000 7f2f7af14640 1 -- 172.21.15.60:0/2776950437 >> 172.21.15.60:0/2776950437 conn(0x7f2f7409b850 msgr2=0x7f2f7409bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:06.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.657+0000 7f2f7af14640 1 -- 172.21.15.60:0/2776950437 shutdown_connections 2024-05-09T05:02:06.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.657+0000 7f2f7af14640 1 -- 172.21.15.60:0/2776950437 wait complete. 2024-05-09T05:02:06.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.659+0000 7f2f7af14640 1 Processor -- start 2024-05-09T05:02:06.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.659+0000 7f2f7af14640 1 -- start start 2024-05-09T05:02:06.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.659+0000 7f2f7af14640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f5c004800 0x7f2f740ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.659+0000 7f2f7af14640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f7409df90 0x7f2f7409e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f7af14640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f741629d0 0x7f2f74162e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f7af14640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7406f7d0 con 0x7f2f7409df90 2024-05-09T05:02:06.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f7af14640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7406f650 con 0x7f2f5c004800 2024-05-09T05:02:06.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f7af14640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f2f740d33f0 con 0x7f2f741629d0 2024-05-09T05:02:06.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f6bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f7409df90 0x7f2f7409e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f6bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f7409df90 0x7f2f7409e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36442/0 (socket says 172.21.15.60:36442) 2024-05-09T05:02:06.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f6bfff640 1 -- 172.21.15.60:0/1912476908 learned_addr learned my addr 172.21.15.60:0/1912476908 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:06.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f794b1640 1 --2- 172.21.15.60:0/1912476908 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f741629d0 0x7f2f74162e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f78cb0640 1 --2- 172.21.15.60:0/1912476908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f5c004800 0x7f2f740ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f6bfff640 1 -- 172.21.15.60:0/1912476908 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f741629d0 msgr2=0x7f2f74162e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f6bfff640 1 --2- 172.21.15.60:0/1912476908 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f741629d0 0x7f2f74162e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f6bfff640 1 -- 172.21.15.60:0/1912476908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f5c004800 msgr2=0x7f2f740ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f6bfff640 1 --2- 172.21.15.60:0/1912476908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f5c004800 0x7f2f740ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f6bfff640 1 -- 172.21.15.60:0/1912476908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7415aa30 con 0x7f2f7409df90 2024-05-09T05:02:06.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f6bfff640 1 --2- 172.21.15.60:0/1912476908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f7409df90 0x7f2f7409e410 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2f64034040 tx=0x7f2f64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:06.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f697fa640 1 -- 172.21.15.60:0/1912476908 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f64041020 con 0x7f2f7409df90 2024-05-09T05:02:06.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f697fa640 1 -- 172.21.15.60:0/1912476908 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f6403c470 con 0x7f2f7409df90 2024-05-09T05:02:06.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f7af14640 1 -- 172.21.15.60:0/1912476908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f741e8b60 con 0x7f2f7409df90 2024-05-09T05:02:06.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f697fa640 1 -- 172.21.15.60:0/1912476908 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f64036040 con 0x7f2f7409df90 2024-05-09T05:02:06.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.660+0000 7f2f7af14640 1 -- 172.21.15.60:0/1912476908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f741e9120 con 0x7f2f7409df90 2024-05-09T05:02:06.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.662+0000 7f2f7af14640 1 -- 172.21.15.60:0/1912476908 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f74051040 con 0x7f2f7409df90 2024-05-09T05:02:06.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.662+0000 7f2f697fa640 1 -- 172.21.15.60:0/1912476908 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2f64044020 con 0x7f2f7409df90 2024-05-09T05:02:06.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.662+0000 7f2f697fa640 1 --2- 172.21.15.60:0/1912476908 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2f60076300 0x7f2f600787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.663+0000 7f2f697fa640 1 -- 172.21.15.60:0/1912476908 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7f2f640b72f0 con 0x7f2f7409df90 2024-05-09T05:02:06.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.663+0000 7f2f78cb0640 1 --2- 172.21.15.60:0/1912476908 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2f60076300 0x7f2f600787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.663+0000 7f2f78cb0640 1 --2- 172.21.15.60:0/1912476908 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2f60076300 0x7f2f600787c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2f70007da0 tx=0x7f2f70001e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:06.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.667+0000 7f2f697fa640 1 -- 172.21.15.60:0/1912476908 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2f640806d0 con 0x7f2f7409df90 2024-05-09T05:02:06.857 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.857+0000 7f2f7af14640 1 -- 172.21.15.60:0/1912476908 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f2f741e8910 con 0x7f2f60076300 2024-05-09T05:02:06.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.857+0000 7f2f697fa640 1 -- 172.21.15.60:0/1912476908 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2f741e8910 con 0x7f2f60076300 2024-05-09T05:02:06.858 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 2 to deep-scrub 2024-05-09T05:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.859+0000 7f2f7af14640 1 -- 172.21.15.60:0/1912476908 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2f60076300 msgr2=0x7f2f600787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.859+0000 7f2f7af14640 1 --2- 172.21.15.60:0/1912476908 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2f60076300 0x7f2f600787c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2f70007da0 tx=0x7f2f70001e70 comp rx=0 tx=0).stop 2024-05-09T05:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.859+0000 7f2f7af14640 1 -- 172.21.15.60:0/1912476908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f7409df90 msgr2=0x7f2f7409e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.859+0000 7f2f7af14640 1 --2- 172.21.15.60:0/1912476908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f7409df90 0x7f2f7409e410 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2f64034040 tx=0x7f2f64004730 comp rx=0 tx=0).stop 2024-05-09T05:02:06.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.859+0000 7f2f7af14640 1 -- 172.21.15.60:0/1912476908 shutdown_connections 2024-05-09T05:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.859+0000 7f2f7af14640 1 --2- 172.21.15.60:0/1912476908 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f2f60076300 0x7f2f600787c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.859+0000 7f2f7af14640 1 --2- 172.21.15.60:0/1912476908 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f2f741629d0 0x7f2f74162e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.859+0000 7f2f7af14640 1 --2- 172.21.15.60:0/1912476908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f7409df90 0x7f2f7409e410 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.859+0000 7f2f7af14640 1 --2- 172.21.15.60:0/1912476908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f5c004800 0x7f2f740ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.859+0000 7f2f7af14640 1 -- 172.21.15.60:0/1912476908 >> 172.21.15.60:0/1912476908 conn(0x7f2f7409b850 msgr2=0x7f2f74099da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.859+0000 7f2f7af14640 1 -- 172.21.15.60:0/1912476908 shutdown_connections 2024-05-09T05:02:06.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.859+0000 7f2f7af14640 1 -- 172.21.15.60:0/1912476908 wait complete. 2024-05-09T05:02:06.871 INFO:tasks.ceph:Scrubbing osd.3 2024-05-09T05:02:06.871 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-05-09T05:02:06.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.978+0000 7f87a3724640 1 Processor -- start 2024-05-09T05:02:06.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.979+0000 7f87a3724640 1 -- start start 2024-05-09T05:02:06.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.979+0000 7f87a3724640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f879c148120 0x7f879c0d2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.979+0000 7f87a3724640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f879c163580 0x7f879c0b5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.979+0000 7f87a3724640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f879c0d31b0 con 0x7f879c0d2ca0 2024-05-09T05:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.979+0000 7f87a3724640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f879c0d3490 con 0x7f879c148120 2024-05-09T05:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.979+0000 7f87a3724640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f879c0b5650 con 0x7f879c163580 2024-05-09T05:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.980+0000 7f87a14c0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f879c148120 0x7f879c0d2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.980+0000 7f87a14c0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f879c148120 0x7f879c0d2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36446/0 (socket says 172.21.15.60:36446) 2024-05-09T05:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.980+0000 7f87a14c0640 1 -- 172.21.15.60:0/2268664244 learned_addr learned my addr 172.21.15.60:0/2268664244 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:06.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.980+0000 7f87a1cc1640 1 --2- 172.21.15.60:0/2268664244 >> v2:172.21.15.114:3300/0 conn(0x7f879c163580 0x7f879c0b5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.980+0000 7f87a14c0640 1 -- 172.21.15.60:0/2268664244 >> v2:172.21.15.114:3300/0 conn(0x7f879c163580 msgr2=0x7f879c0b5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.980+0000 7f87a14c0640 1 --2- 172.21.15.60:0/2268664244 >> v2:172.21.15.114:3300/0 conn(0x7f879c163580 0x7f879c0b5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.980+0000 7f87a14c0640 1 -- 172.21.15.60:0/2268664244 >> v1:172.21.15.114:6789/0 conn(0x7f879c0d2ca0 legacy=0x7f879c162e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.980+0000 7f87a14c0640 1 -- 172.21.15.60:0/2268664244 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f879c0b5930 con 0x7f879c148120 2024-05-09T05:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.981+0000 7f87a14c0640 1 --2- 172.21.15.60:0/2268664244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c148120 0x7f879c0d2760 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f879c0d3bb0 tx=0x7f879802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c2495edbb4d33da server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.981+0000 7f8793fff640 1 -- 172.21.15.60:0/2268664244 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8798037070 con 0x7f879c148120 2024-05-09T05:02:06.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.981+0000 7f8793fff640 1 -- 172.21.15.60:0/2268664244 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8798032440 con 0x7f879c148120 2024-05-09T05:02:06.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.981+0000 7f8793fff640 1 -- 172.21.15.60:0/2268664244 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8798032810 con 0x7f879c148120 2024-05-09T05:02:06.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.981+0000 7f87a3724640 1 -- 172.21.15.60:0/2268664244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c148120 msgr2=0x7f879c0d2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.981+0000 7f87a3724640 1 --2- 172.21.15.60:0/2268664244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c148120 0x7f879c0d2760 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f879c0d3bb0 tx=0x7f879802a270 comp rx=0 tx=0).stop 2024-05-09T05:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.982+0000 7f87a3724640 1 -- 172.21.15.60:0/2268664244 shutdown_connections 2024-05-09T05:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.982+0000 7f87a3724640 1 --2- 172.21.15.60:0/2268664244 >> v2:172.21.15.114:3300/0 conn(0x7f879c163580 0x7f879c0b5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.982+0000 7f87a3724640 1 --2- 172.21.15.60:0/2268664244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c148120 0x7f879c0d2760 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.982+0000 7f87a3724640 1 -- 172.21.15.60:0/2268664244 >> 172.21.15.60:0/2268664244 conn(0x7f879c06f640 msgr2=0x7f879c073840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.982+0000 7f87a3724640 1 -- 172.21.15.60:0/2268664244 shutdown_connections 2024-05-09T05:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.982+0000 7f87a3724640 1 -- 172.21.15.60:0/2268664244 wait complete. 2024-05-09T05:02:06.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.983+0000 7f87a3724640 1 Processor -- start 2024-05-09T05:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.983+0000 7f87a3724640 1 -- start start 2024-05-09T05:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f87a3724640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c0d2ca0 0x7f879c13e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f87a3724640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f879c148120 0x7f879c1f87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f87a3724640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f879c163580 0x7f879c1f8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f87a3724640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f879c163cb0 con 0x7f879c0d2ca0 2024-05-09T05:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f87a3724640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f879c163b30 con 0x7f879c148120 2024-05-09T05:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f87a3724640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f879c0b6390 con 0x7f879c163580 2024-05-09T05:02:06.988 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f87a14c0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c0d2ca0 0x7f879c13e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f87a14c0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c0d2ca0 0x7f879c13e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36456/0 (socket says 172.21.15.60:36456) 2024-05-09T05:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f87a14c0640 1 -- 172.21.15.60:0/991057189 learned_addr learned my addr 172.21.15.60:0/991057189 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f87a1cc1640 1 --2- 172.21.15.60:0/991057189 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f879c163580 0x7f879c1f8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f87a0cbf640 1 --2- 172.21.15.60:0/991057189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f879c148120 0x7f879c1f87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f87a14c0640 1 -- 172.21.15.60:0/991057189 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f879c163580 msgr2=0x7f879c1f8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f87a14c0640 1 --2- 172.21.15.60:0/991057189 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f879c163580 0x7f879c1f8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f87a14c0640 1 -- 172.21.15.60:0/991057189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f879c148120 msgr2=0x7f879c1f87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:06.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f87a14c0640 1 --2- 172.21.15.60:0/991057189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f879c148120 0x7f879c1f87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f87a14c0640 1 -- 172.21.15.60:0/991057189 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f879c1622b0 con 0x7f879c0d2ca0 2024-05-09T05:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f87a14c0640 1 --2- 172.21.15.60:0/991057189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c0d2ca0 0x7f879c13e630 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f879802a2b0 tx=0x7f87980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f8791ffb640 1 -- 172.21.15.60:0/991057189 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8798041020 con 0x7f879c0d2ca0 2024-05-09T05:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f8791ffb640 1 -- 172.21.15.60:0/991057189 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87980306c0 con 0x7f879c0d2ca0 2024-05-09T05:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.984+0000 7f87a3724640 1 -- 172.21.15.60:0/991057189 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f879c13eb70 con 0x7f879c0d2ca0 2024-05-09T05:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.985+0000 7f8791ffb640 1 -- 172.21.15.60:0/991057189 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8798037040 con 0x7f879c0d2ca0 2024-05-09T05:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.985+0000 7f87a3724640 1 -- 172.21.15.60:0/991057189 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f879c1617d0 con 0x7f879c0d2ca0 2024-05-09T05:02:06.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.985+0000 7f87a3724640 1 -- 172.21.15.60:0/991057189 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f879c051040 con 0x7f879c0d2ca0 2024-05-09T05:02:06.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.986+0000 7f8791ffb640 1 -- 172.21.15.60:0/991057189 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8798044020 con 0x7f879c0d2ca0 2024-05-09T05:02:06.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.987+0000 7f8791ffb640 1 --2- 172.21.15.60:0/991057189 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f87800762d0 0x7f8780078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.987+0000 7f8791ffb640 1 -- 172.21.15.60:0/991057189 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7f87980b8180 con 0x7f879c0d2ca0 2024-05-09T05:02:06.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.987+0000 7f8791ffb640 1 --2- 172.21.15.60:0/991057189 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x7f878007c770 0x7f878007ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.987+0000 7f8791ffb640 1 -- 172.21.15.60:0/991057189 --> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f879803fe50 con 0x7f878007c770 2024-05-09T05:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.987+0000 7f8791ffb640 1 -- 172.21.15.60:0/991057189 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f87980b8530 con 0x7f879c0d2ca0 2024-05-09T05:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.987+0000 7f87a1cc1640 1 --2- 172.21.15.60:0/991057189 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x7f878007c770 0x7f878007ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.987+0000 7f87a0cbf640 1 --2- 172.21.15.60:0/991057189 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f87800762d0 0x7f8780078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.988+0000 7f87a1cc1640 1 --2- 172.21.15.60:0/991057189 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x7f878007c770 0x7f878007ebb0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.988+0000 7f87a0cbf640 1 --2- 172.21.15.60:0/991057189 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f87800762d0 0x7f8780078790 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f879c162b80 tx=0x7f878c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:06.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:06.988+0000 7f8791ffb640 1 -- 172.21.15.60:0/991057189 <== osd.3 v2:172.21.15.60:6824/1069806621 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f879803fe50 con 0x7f878007c770 2024-05-09T05:02:07.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.002+0000 7f87a3724640 1 -- 172.21.15.60:0/991057189 --> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f879c162be0 con 0x7f878007c770 2024-05-09T05:02:07.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.002+0000 7f8791ffb640 1 -- 172.21.15.60:0/991057189 <== osd.3 v2:172.21.15.60:6824/1069806621 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f879c162be0 con 0x7f878007c770 2024-05-09T05:02:07.003 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:02:07.003 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T05:02:07.003 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:02:07.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.002+0000 7f87a3724640 1 -- 172.21.15.60:0/991057189 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x7f878007c770 msgr2=0x7f878007ebb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.002+0000 7f87a3724640 1 --2- 172.21.15.60:0/991057189 >> [v2:172.21.15.60:6824/1069806621,v1:172.21.15.60:6825/1069806621] conn(0x7f878007c770 0x7f878007ebb0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.002+0000 7f87a3724640 1 -- 172.21.15.60:0/991057189 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f87800762d0 msgr2=0x7f8780078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.002+0000 7f87a3724640 1 --2- 172.21.15.60:0/991057189 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f87800762d0 0x7f8780078790 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f879c162b80 tx=0x7f878c009040 comp rx=0 tx=0).stop 2024-05-09T05:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.002+0000 7f87a3724640 1 -- 172.21.15.60:0/991057189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c0d2ca0 msgr2=0x7f879c13e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.002+0000 7f87a3724640 1 --2- 172.21.15.60:0/991057189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c0d2ca0 0x7f879c13e630 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f879802a2b0 tx=0x7f87980046c0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.002+0000 7f87a1cc1640 1 -- 172.21.15.60:0/991057189 reap_dead start 2024-05-09T05:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.002+0000 7f87a3724640 1 -- 172.21.15.60:0/991057189 shutdown_connections 2024-05-09T05:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.002+0000 7f87a3724640 1 -- 172.21.15.60:0/991057189 >> 172.21.15.60:0/991057189 conn(0x7f879c06f640 msgr2=0x7f879c15eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:07.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.002+0000 7f87a3724640 1 -- 172.21.15.60:0/991057189 shutdown_connections 2024-05-09T05:02:07.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.002+0000 7f87a3724640 1 -- 172.21.15.60:0/991057189 wait complete. 2024-05-09T05:02:07.014 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-05-09T05:02:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.118+0000 7f1ae6140640 1 Processor -- start 2024-05-09T05:02:07.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.118+0000 7f1ae6140640 1 -- start start 2024-05-09T05:02:07.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.119+0000 7f1ae6140640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1ae0098af0 0x7f1ae009b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.119+0000 7f1ae6140640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1ae00d0070 0x7f1ae00b0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.119+0000 7f1ae6140640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ae00d05e0 con 0x7f1ae0098ed0 2024-05-09T05:02:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.119+0000 7f1ae6140640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f1ae00ae9b0 con 0x7f1ae0098af0 2024-05-09T05:02:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.119+0000 7f1ae6140640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f1ae00aec90 con 0x7f1ae00d0070 2024-05-09T05:02:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.119+0000 7f1adf7fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1ae0098af0 0x7f1ae009b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.119+0000 7f1adeffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1ae0098ed0 0x7f1ae00b2dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34280/0 (socket says 172.21.15.60:34280) 2024-05-09T05:02:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.119+0000 7f1adeffd640 1 -- 172.21.15.60:0/2920720001 learned_addr learned my addr 172.21.15.60:0/2920720001 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:02:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.119+0000 7f1adffff640 1 --2- 172.21.15.60:0/2920720001 >> v2:172.21.15.114:3300/0 conn(0x7f1ae00d0070 0x7f1ae00b0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.119+0000 7f1adf7fe640 1 -- 172.21.15.60:0/2920720001 >> v2:172.21.15.114:3300/0 conn(0x7f1ae00d0070 msgr2=0x7f1ae00b0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.119+0000 7f1adf7fe640 1 --2- 172.21.15.60:0/2920720001 >> v2:172.21.15.114:3300/0 conn(0x7f1ae00d0070 0x7f1ae00b0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.119+0000 7f1adf7fe640 1 -- 172.21.15.60:0/2920720001 >> v1:172.21.15.60:6789/0 conn(0x7f1ae0098ed0 legacy=0x7f1ae00b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.119+0000 7f1adf7fe640 1 -- 172.21.15.60:0/2920720001 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae00aef70 con 0x7f1ae0098af0 2024-05-09T05:02:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.120+0000 7f1adf7fe640 1 --2- 172.21.15.60:0/2920720001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae0098af0 0x7f1ae009b3a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1ae01642b0 tx=0x7f1ad002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa3c505b32bd780 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.120+0000 7f1ade7fc640 1 -- 172.21.15.60:0/2920720001 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0037070 con 0x7f1ae0098af0 2024-05-09T05:02:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.120+0000 7f1ade7fc640 1 -- 172.21.15.60:0/2920720001 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ad0032440 con 0x7f1ae0098af0 2024-05-09T05:02:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.120+0000 7f1ade7fc640 1 -- 172.21.15.60:0/2920720001 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0032810 con 0x7f1ae0098af0 2024-05-09T05:02:07.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.121+0000 7f1ae6140640 1 -- 172.21.15.60:0/2920720001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae0098af0 msgr2=0x7f1ae009b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.121+0000 7f1ae6140640 1 --2- 172.21.15.60:0/2920720001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae0098af0 0x7f1ae009b3a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1ae01642b0 tx=0x7f1ad002a1a0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.121+0000 7f1ae6140640 1 -- 172.21.15.60:0/2920720001 shutdown_connections 2024-05-09T05:02:07.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.121+0000 7f1ae6140640 1 --2- 172.21.15.60:0/2920720001 >> v2:172.21.15.114:3300/0 conn(0x7f1ae00d0070 0x7f1ae00b0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.121+0000 7f1ae6140640 1 --2- 172.21.15.60:0/2920720001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae0098af0 0x7f1ae009b3a0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.121+0000 7f1ae6140640 1 -- 172.21.15.60:0/2920720001 >> 172.21.15.60:0/2920720001 conn(0x7f1ae00b42e0 msgr2=0x7f1ae00b46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.121+0000 7f1ae6140640 1 -- 172.21.15.60:0/2920720001 shutdown_connections 2024-05-09T05:02:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.121+0000 7f1ae6140640 1 -- 172.21.15.60:0/2920720001 wait complete. 2024-05-09T05:02:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.122+0000 7f1ae6140640 1 Processor -- start 2024-05-09T05:02:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.122+0000 7f1ae6140640 1 -- start start 2024-05-09T05:02:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.123+0000 7f1ae6140640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ae0098af0 0x7f1ae01f4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.123+0000 7f1ae6140640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae0098ed0 0x7f1ae01f60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.123+0000 7f1ae6140640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ae00d0070 0x7f1ae00ace80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.123+0000 7f1ae6140640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae00b1960 con 0x7f1ae0098ed0 2024-05-09T05:02:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.123+0000 7f1ae6140640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae00b17e0 con 0x7f1ae00d0070 2024-05-09T05:02:07.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.123+0000 7f1ae6140640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1ae009b8e0 con 0x7f1ae0098af0 2024-05-09T05:02:07.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.123+0000 7f1adeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae0098ed0 0x7f1ae01f60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.123+0000 7f1adeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae0098ed0 0x7f1ae01f60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36474/0 (socket says 172.21.15.60:36474) 2024-05-09T05:02:07.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.123+0000 7f1adeffd640 1 -- 172.21.15.60:0/1902395912 learned_addr learned my addr 172.21.15.60:0/1902395912 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:07.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.123+0000 7f1adeffd640 1 -- 172.21.15.60:0/1902395912 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ae0098af0 msgr2=0x7f1ae01f4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.123+0000 7f1adeffd640 1 --2- 172.21.15.60:0/1902395912 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ae0098af0 0x7f1ae01f4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.123+0000 7f1adeffd640 1 -- 172.21.15.60:0/1902395912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ae00d0070 msgr2=0x7f1ae00ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.123+0000 7f1adeffd640 1 --2- 172.21.15.60:0/1902395912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ae00d0070 0x7f1ae00ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.123+0000 7f1adeffd640 1 -- 172.21.15.60:0/1902395912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae00aae60 con 0x7f1ae0098ed0 2024-05-09T05:02:07.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.124+0000 7f1adeffd640 1 --2- 172.21.15.60:0/1902395912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae0098ed0 0x7f1ae01f60d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1ad400c4d0 tx=0x7f1ad400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.124+0000 7f1ac7fff640 1 -- 172.21.15.60:0/1902395912 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4017070 con 0x7f1ae0098ed0 2024-05-09T05:02:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.124+0000 7f1ac7fff640 1 -- 172.21.15.60:0/1902395912 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ad4005d00 con 0x7f1ae0098ed0 2024-05-09T05:02:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.124+0000 7f1ae6140640 1 -- 172.21.15.60:0/1902395912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ae00ab140 con 0x7f1ae0098ed0 2024-05-09T05:02:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.124+0000 7f1ac7fff640 1 -- 172.21.15.60:0/1902395912 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad40072b0 con 0x7f1ae0098ed0 2024-05-09T05:02:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.124+0000 7f1ae6140640 1 -- 172.21.15.60:0/1902395912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ae00ab700 con 0x7f1ae0098ed0 2024-05-09T05:02:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.125+0000 7f1ae6140640 1 -- 172.21.15.60:0/1902395912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ae0051040 con 0x7f1ae0098ed0 2024-05-09T05:02:07.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.126+0000 7f1ac7fff640 1 -- 172.21.15.60:0/1902395912 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1ad401a070 con 0x7f1ae0098ed0 2024-05-09T05:02:07.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.126+0000 7f1ac7fff640 1 --2- 172.21.15.60:0/1902395912 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1acc0762d0 0x7f1acc078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.126+0000 7f1ac7fff640 1 -- 172.21.15.60:0/1902395912 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7f1ad40967c0 con 0x7f1ae0098ed0 2024-05-09T05:02:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.126+0000 7f1adf7fe640 1 --2- 172.21.15.60:0/1902395912 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1acc0762d0 0x7f1acc078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.127+0000 7f1adf7fe640 1 --2- 172.21.15.60:0/1902395912 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1acc0762d0 0x7f1acc078790 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0034000 tx=0x7f1ad002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.130+0000 7f1ac7fff640 1 -- 172.21.15.60:0/1902395912 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1ad4060290 con 0x7f1ae0098ed0 2024-05-09T05:02:07.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.320+0000 7f1ae6140640 1 -- 172.21.15.60:0/1902395912 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f1ae009ada0 con 0x7f1acc0762d0 2024-05-09T05:02:07.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.320+0000 7f1ac7fff640 1 -- 172.21.15.60:0/1902395912 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1ae009ada0 con 0x7f1acc0762d0 2024-05-09T05:02:07.322 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 3 to deep-scrub 2024-05-09T05:02:07.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.322+0000 7f1ae6140640 1 -- 172.21.15.60:0/1902395912 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1acc0762d0 msgr2=0x7f1acc078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.322+0000 7f1ae6140640 1 --2- 172.21.15.60:0/1902395912 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1acc0762d0 0x7f1acc078790 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0034000 tx=0x7f1ad002c000 comp rx=0 tx=0).stop 2024-05-09T05:02:07.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.322+0000 7f1ae6140640 1 -- 172.21.15.60:0/1902395912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae0098ed0 msgr2=0x7f1ae01f60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.322+0000 7f1ae6140640 1 --2- 172.21.15.60:0/1902395912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae0098ed0 0x7f1ae01f60d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1ad400c4d0 tx=0x7f1ad400c990 comp rx=0 tx=0).stop 2024-05-09T05:02:07.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.322+0000 7f1ae6140640 1 -- 172.21.15.60:0/1902395912 shutdown_connections 2024-05-09T05:02:07.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.322+0000 7f1ae6140640 1 --2- 172.21.15.60:0/1902395912 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1acc0762d0 0x7f1acc078790 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.322+0000 7f1ae6140640 1 --2- 172.21.15.60:0/1902395912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ae00d0070 0x7f1ae00ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.322+0000 7f1ae6140640 1 --2- 172.21.15.60:0/1902395912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ae0098ed0 0x7f1ae01f60d0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.322+0000 7f1ae6140640 1 --2- 172.21.15.60:0/1902395912 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1ae0098af0 0x7f1ae01f4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.322+0000 7f1ae6140640 1 -- 172.21.15.60:0/1902395912 >> 172.21.15.60:0/1902395912 conn(0x7f1ae00b42e0 msgr2=0x7f1ae015fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:07.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.322+0000 7f1ae6140640 1 -- 172.21.15.60:0/1902395912 shutdown_connections 2024-05-09T05:02:07.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.323+0000 7f1ae6140640 1 -- 172.21.15.60:0/1902395912 wait complete. 2024-05-09T05:02:07.334 INFO:tasks.ceph:Scrubbing osd.4 2024-05-09T05:02:07.334 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-05-09T05:02:07.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.441+0000 7f7ec6a1d640 1 Processor -- start 2024-05-09T05:02:07.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.441+0000 7f7ec6a1d640 1 -- start start 2024-05-09T05:02:07.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.442+0000 7f7ec6a1d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7ec01481b0 0x7f7ec00b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.442+0000 7f7ec6a1d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ec00b5a90 con 0x7f7ec00d4cc0 2024-05-09T05:02:07.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.442+0000 7f7ec6a1d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ec00b5d70 con 0x7f7ec0148710 2024-05-09T05:02:07.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.442+0000 7f7ec6a1d640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f7ec00b6050 con 0x7f7ec01481b0 2024-05-09T05:02:07.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.442+0000 7f7ebffff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7ec01481b0 0x7f7ec00b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.442+0000 7f7ec4fba640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7ec00d4cc0 0x7f7ec00d1d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34292/0 (socket says 172.21.15.60:34292) 2024-05-09T05:02:07.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.442+0000 7f7ec4fba640 1 -- 172.21.15.60:0/2305755939 learned_addr learned my addr 172.21.15.60:0/2305755939 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:02:07.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.443+0000 7f7ebffff640 1 -- 172.21.15.60:0/2305755939 >> v1:172.21.15.114:6789/0 conn(0x7f7ec0148710 legacy=0x7f7ec00d45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.443+0000 7f7ebffff640 1 -- 172.21.15.60:0/2305755939 >> v1:172.21.15.60:6789/0 conn(0x7f7ec00d4cc0 legacy=0x7f7ec00d1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.443+0000 7f7ebffff640 1 -- 172.21.15.60:0/2305755939 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec00b6330 con 0x7f7ec01481b0 2024-05-09T05:02:07.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.443+0000 7f7ebeffd640 1 -- 172.21.15.60:0/2305755939 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1231404944 0 0) 0x7f7ec00b5a90 con 0x7f7ec00d4cc0 2024-05-09T05:02:07.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.443+0000 7f7ebffff640 1 --2- 172.21.15.60:0/2305755939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ec01481b0 0x7f7ec00b54b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7eb00097a0 tx=0x7f7eb002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=684f094c989c88da server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:07.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.444+0000 7f7ebeffd640 1 -- 172.21.15.60:0/2305755939 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb0037070 con 0x7f7ec01481b0 2024-05-09T05:02:07.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.444+0000 7f7ebeffd640 1 -- 172.21.15.60:0/2305755939 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7eb0032440 con 0x7f7ec01481b0 2024-05-09T05:02:07.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.444+0000 7f7ebeffd640 1 -- 172.21.15.60:0/2305755939 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb0032810 con 0x7f7ec01481b0 2024-05-09T05:02:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.444+0000 7f7ec6a1d640 1 -- 172.21.15.60:0/2305755939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ec01481b0 msgr2=0x7f7ec00b54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.444+0000 7f7ec6a1d640 1 --2- 172.21.15.60:0/2305755939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ec01481b0 0x7f7ec00b54b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7eb00097a0 tx=0x7f7eb002a1a0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.444+0000 7f7ec6a1d640 1 -- 172.21.15.60:0/2305755939 shutdown_connections 2024-05-09T05:02:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.444+0000 7f7ec6a1d640 1 --2- 172.21.15.60:0/2305755939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ec01481b0 0x7f7ec00b54b0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.444+0000 7f7ec6a1d640 1 -- 172.21.15.60:0/2305755939 >> 172.21.15.60:0/2305755939 conn(0x7f7ec006f640 msgr2=0x7f7ec0073840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.444+0000 7f7ec6a1d640 1 -- 172.21.15.60:0/2305755939 shutdown_connections 2024-05-09T05:02:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.444+0000 7f7ec6a1d640 1 -- 172.21.15.60:0/2305755939 wait complete. 2024-05-09T05:02:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.445+0000 7f7ec6a1d640 1 Processor -- start 2024-05-09T05:02:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.446+0000 7f7ec6a1d640 1 -- start start 2024-05-09T05:02:07.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.446+0000 7f7ec6a1d640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ec00d4cc0 0x7f7ec0159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.446+0000 7f7ec6a1d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ec01481b0 0x7f7ec015bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.446+0000 7f7ec6a1d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ec0148710 0x7f7ec015a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.446+0000 7f7ec6a1d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec00d25a0 con 0x7f7ec0148710 2024-05-09T05:02:07.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.446+0000 7f7ec6a1d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec00d2420 con 0x7f7ec01481b0 2024-05-09T05:02:07.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.446+0000 7f7ec6a1d640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f7ec00d2720 con 0x7f7ec00d4cc0 2024-05-09T05:02:07.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.446+0000 7f7ec4fba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ec0148710 0x7f7ec015a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.446+0000 7f7ec4fba640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ec0148710 0x7f7ec015a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36486/0 (socket says 172.21.15.60:36486) 2024-05-09T05:02:07.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.446+0000 7f7ec4fba640 1 -- 172.21.15.60:0/1412130386 learned_addr learned my addr 172.21.15.60:0/1412130386 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:07.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.447+0000 7f7ebf7fe640 1 --2- 172.21.15.60:0/1412130386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ec01481b0 0x7f7ec015bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.447+0000 7f7ebffff640 1 --2- 172.21.15.60:0/1412130386 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ec00d4cc0 0x7f7ec0159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.447+0000 7f7ec4fba640 1 -- 172.21.15.60:0/1412130386 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ec00d4cc0 msgr2=0x7f7ec0159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.447+0000 7f7ec4fba640 1 --2- 172.21.15.60:0/1412130386 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f7ec00d4cc0 0x7f7ec0159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.447+0000 7f7ec4fba640 1 -- 172.21.15.60:0/1412130386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ec01481b0 msgr2=0x7f7ec015bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.447+0000 7f7ec4fba640 1 --2- 172.21.15.60:0/1412130386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ec01481b0 0x7f7ec015bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.447+0000 7f7ec4fba640 1 -- 172.21.15.60:0/1412130386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec015aba0 con 0x7f7ec0148710 2024-05-09T05:02:07.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.447+0000 7f7ebf7fe640 1 --2- 172.21.15.60:0/1412130386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ec01481b0 0x7f7ec015bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:02:07.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.447+0000 7f7ec4fba640 1 --2- 172.21.15.60:0/1412130386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ec0148710 0x7f7ec015a450 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f7eb80015a0 tx=0x7f7eb800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:07.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.447+0000 7f7ebcff9640 1 -- 172.21.15.60:0/1412130386 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8018070 con 0x7f7ec0148710 2024-05-09T05:02:07.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.447+0000 7f7ebcff9640 1 -- 172.21.15.60:0/1412130386 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7eb800ce50 con 0x7f7ec0148710 2024-05-09T05:02:07.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.447+0000 7f7ec6a1d640 1 -- 172.21.15.60:0/1412130386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ec015ae80 con 0x7f7ec0148710 2024-05-09T05:02:07.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.447+0000 7f7ebcff9640 1 -- 172.21.15.60:0/1412130386 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb800a820 con 0x7f7ec0148710 2024-05-09T05:02:07.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.447+0000 7f7ec6a1d640 1 -- 172.21.15.60:0/1412130386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ec015b440 con 0x7f7ec0148710 2024-05-09T05:02:07.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.448+0000 7f7ec6a1d640 1 -- 172.21.15.60:0/1412130386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7ec015b820 con 0x7f7ec0148710 2024-05-09T05:02:07.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.449+0000 7f7ebcff9640 1 -- 172.21.15.60:0/1412130386 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7eb801b070 con 0x7f7ec0148710 2024-05-09T05:02:07.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.449+0000 7f7ebcff9640 1 --2- 172.21.15.60:0/1412130386 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7eac0762d0 0x7f7eac078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.449+0000 7f7ebcff9640 1 -- 172.21.15.60:0/1412130386 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7f7eb8096a00 con 0x7f7ec0148710 2024-05-09T05:02:07.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.450+0000 7f7ebffff640 1 --2- 172.21.15.60:0/1412130386 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7eac0762d0 0x7f7eac078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.450+0000 7f7ebcff9640 1 --2- 172.21.15.60:0/1412130386 >> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] conn(0x7f7eac07c770 0x7f7eac07ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.450+0000 7f7ebcff9640 1 -- 172.21.15.60:0/1412130386 --> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7eac07f280 con 0x7f7eac07c770 2024-05-09T05:02:07.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.450+0000 7f7ebcff9640 1 -- 172.21.15.60:0/1412130386 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f7eb809b2a0 con 0x7f7ec0148710 2024-05-09T05:02:07.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.450+0000 7f7ebf7fe640 1 --2- 172.21.15.60:0/1412130386 >> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] conn(0x7f7eac07c770 0x7f7eac07ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.450+0000 7f7ebffff640 1 --2- 172.21.15.60:0/1412130386 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7eac0762d0 0x7f7eac078790 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0034000 tx=0x7f7eb002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:07.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.450+0000 7f7ebf7fe640 1 --2- 172.21.15.60:0/1412130386 >> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] conn(0x7f7eac07c770 0x7f7eac07ebb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:07.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.451+0000 7f7ebcff9640 1 -- 172.21.15.60:0/1412130386 <== osd.4 v2:172.21.15.114:6801/3462725972 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7eac07f280 con 0x7f7eac07c770 2024-05-09T05:02:07.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.464+0000 7f7ec6a1d640 1 -- 172.21.15.60:0/1412130386 --> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7ec0051040 con 0x7f7eac07c770 2024-05-09T05:02:07.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.465+0000 7f7ebcff9640 1 -- 172.21.15.60:0/1412130386 <== osd.4 v2:172.21.15.114:6801/3462725972 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7ec0051040 con 0x7f7eac07c770 2024-05-09T05:02:07.466 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:02:07.466 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T05:02:07.467 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:02:07.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.465+0000 7f7ec6a1d640 1 -- 172.21.15.60:0/1412130386 >> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] conn(0x7f7eac07c770 msgr2=0x7f7eac07ebb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.465+0000 7f7ec6a1d640 1 --2- 172.21.15.60:0/1412130386 >> [v2:172.21.15.114:6801/3462725972,v1:172.21.15.114:6803/3462725972] conn(0x7f7eac07c770 0x7f7eac07ebb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.465+0000 7f7ec6a1d640 1 -- 172.21.15.60:0/1412130386 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7eac0762d0 msgr2=0x7f7eac078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.465+0000 7f7ec6a1d640 1 --2- 172.21.15.60:0/1412130386 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f7eac0762d0 0x7f7eac078790 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0034000 tx=0x7f7eb002e000 comp rx=0 tx=0).stop 2024-05-09T05:02:07.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.465+0000 7f7ec6a1d640 1 -- 172.21.15.60:0/1412130386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ec0148710 msgr2=0x7f7ec015a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.465+0000 7f7ec6a1d640 1 --2- 172.21.15.60:0/1412130386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ec0148710 0x7f7ec015a450 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f7eb80015a0 tx=0x7f7eb800cc00 comp rx=0 tx=0).stop 2024-05-09T05:02:07.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.465+0000 7f7ec4fba640 1 -- 172.21.15.60:0/1412130386 reap_dead start 2024-05-09T05:02:07.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.465+0000 7f7ec6a1d640 1 -- 172.21.15.60:0/1412130386 shutdown_connections 2024-05-09T05:02:07.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.465+0000 7f7ec6a1d640 1 -- 172.21.15.60:0/1412130386 >> 172.21.15.60:0/1412130386 conn(0x7f7ec006f640 msgr2=0x7f7ec00b8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:07.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.465+0000 7f7ec6a1d640 1 -- 172.21.15.60:0/1412130386 shutdown_connections 2024-05-09T05:02:07.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.465+0000 7f7ec6a1d640 1 -- 172.21.15.60:0/1412130386 wait complete. 2024-05-09T05:02:07.476 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-05-09T05:02:07.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.580+0000 7f190ee54640 1 Processor -- start 2024-05-09T05:02:07.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.580+0000 7f190ee54640 1 -- start start 2024-05-09T05:02:07.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.581+0000 7f190ee54640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f190815b990 0x7f1908072f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.581+0000 7f190ee54640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1908070990 con 0x7f1908097de0 2024-05-09T05:02:07.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.581+0000 7f190ee54640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1908072630 con 0x7f190809a610 2024-05-09T05:02:07.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.581+0000 7f190ee54640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f190806fd80 con 0x7f190815b990 2024-05-09T05:02:07.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.581+0000 7f190cbf0640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f190815b990 0x7f1908072f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.581+0000 7f1907fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1908097de0 0x7f1908071810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34298/0 (socket says 172.21.15.60:34298) 2024-05-09T05:02:07.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.581+0000 7f1907fff640 1 -- 172.21.15.60:0/120106496 learned_addr learned my addr 172.21.15.60:0/120106496 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:02:07.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.582+0000 7f190cbf0640 1 -- 172.21.15.60:0/120106496 >> v1:172.21.15.114:6789/0 conn(0x7f190809a610 legacy=0x7f1908071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:07.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.582+0000 7f190cbf0640 1 -- 172.21.15.60:0/120106496 >> v1:172.21.15.60:6789/0 conn(0x7f1908097de0 legacy=0x7f1908071810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.582+0000 7f19077fe640 1 -- 172.21.15.60:0/120106496 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2274084431 0 0) 0x7f1908070990 con 0x7f1908097de0 2024-05-09T05:02:07.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.582+0000 7f190cbf0640 1 -- 172.21.15.60:0/120106496 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1908070060 con 0x7f190815b990 2024-05-09T05:02:07.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.582+0000 7f190cbf0640 1 --2- 172.21.15.60:0/120106496 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f190815b990 0x7f1908072f80 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f18f80097a0 tx=0x7f18f802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e14185cfa470263 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:07.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.583+0000 7f19077fe640 1 -- 172.21.15.60:0/120106496 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f8037070 con 0x7f190815b990 2024-05-09T05:02:07.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.583+0000 7f19077fe640 1 -- 172.21.15.60:0/120106496 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18f8032440 con 0x7f190815b990 2024-05-09T05:02:07.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.583+0000 7f19077fe640 1 -- 172.21.15.60:0/120106496 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f8032810 con 0x7f190815b990 2024-05-09T05:02:07.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.583+0000 7f190ee54640 1 -- 172.21.15.60:0/120106496 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f190815b990 msgr2=0x7f1908072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.583+0000 7f190ee54640 1 --2- 172.21.15.60:0/120106496 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f190815b990 0x7f1908072f80 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f18f80097a0 tx=0x7f18f802a180 comp rx=0 tx=0).stop 2024-05-09T05:02:07.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.583+0000 7f190ee54640 1 -- 172.21.15.60:0/120106496 shutdown_connections 2024-05-09T05:02:07.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.583+0000 7f190ee54640 1 --2- 172.21.15.60:0/120106496 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f190815b990 0x7f1908072f80 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.583+0000 7f190ee54640 1 -- 172.21.15.60:0/120106496 >> 172.21.15.60:0/120106496 conn(0x7f19080972f0 msgr2=0x7f19080976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:07.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.583+0000 7f190ee54640 1 -- 172.21.15.60:0/120106496 shutdown_connections 2024-05-09T05:02:07.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.583+0000 7f190ee54640 1 -- 172.21.15.60:0/120106496 wait complete. 2024-05-09T05:02:07.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.584+0000 7f190ee54640 1 Processor -- start 2024-05-09T05:02:07.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.584+0000 7f190ee54640 1 -- start start 2024-05-09T05:02:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.585+0000 7f190ee54640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1908097de0 0x7f19080b1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.585+0000 7f190ee54640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f190809a610 0x7f19081f2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.585+0000 7f190ee54640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f190815b990 0x7f19081e4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.585+0000 7f190ee54640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1908073c00 con 0x7f190809a610 2024-05-09T05:02:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.585+0000 7f190ee54640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1908070620 con 0x7f1908097de0 2024-05-09T05:02:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.585+0000 7f190ee54640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f190806fa20 con 0x7f190815b990 2024-05-09T05:02:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.585+0000 7f1907fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f190809a610 0x7f19081f2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.585+0000 7f1907fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f190809a610 0x7f19081f2d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36516/0 (socket says 172.21.15.60:36516) 2024-05-09T05:02:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.585+0000 7f1907fff640 1 -- 172.21.15.60:0/3881251502 learned_addr learned my addr 172.21.15.60:0/3881251502 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:07.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.586+0000 7f190cbf0640 1 --2- 172.21.15.60:0/3881251502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1908097de0 0x7f19080b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.586+0000 7f190d3f1640 1 --2- 172.21.15.60:0/3881251502 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f190815b990 0x7f19081e4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.586+0000 7f1907fff640 1 -- 172.21.15.60:0/3881251502 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f190815b990 msgr2=0x7f19081e4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.586+0000 7f1907fff640 1 --2- 172.21.15.60:0/3881251502 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f190815b990 0x7f19081e4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.586+0000 7f1907fff640 1 -- 172.21.15.60:0/3881251502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1908097de0 msgr2=0x7f19080b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.586+0000 7f1907fff640 1 --2- 172.21.15.60:0/3881251502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1908097de0 0x7f19080b1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.586+0000 7f1907fff640 1 -- 172.21.15.60:0/3881251502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19081e7930 con 0x7f190809a610 2024-05-09T05:02:07.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.586+0000 7f1907fff640 1 --2- 172.21.15.60:0/3881251502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f190809a610 0x7f19081f2d20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f18fc00c770 tx=0x7f18fc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:07.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.586+0000 7f19057fa640 1 -- 172.21.15.60:0/3881251502 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18fc018070 con 0x7f190809a610 2024-05-09T05:02:07.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.586+0000 7f19057fa640 1 -- 172.21.15.60:0/3881251502 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18fc00a640 con 0x7f190809a610 2024-05-09T05:02:07.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.586+0000 7f190ee54640 1 -- 172.21.15.60:0/3881251502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19081e7bd0 con 0x7f190809a610 2024-05-09T05:02:07.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.586+0000 7f19057fa640 1 -- 172.21.15.60:0/3881251502 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18fc00aa30 con 0x7f190809a610 2024-05-09T05:02:07.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.586+0000 7f190ee54640 1 -- 172.21.15.60:0/3881251502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19081e8190 con 0x7f190809a610 2024-05-09T05:02:07.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.587+0000 7f190ee54640 1 -- 172.21.15.60:0/3881251502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1908051040 con 0x7f190809a610 2024-05-09T05:02:07.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.588+0000 7f19057fa640 1 -- 172.21.15.60:0/3881251502 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f18fc01b070 con 0x7f190809a610 2024-05-09T05:02:07.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.588+0000 7f19057fa640 1 --2- 172.21.15.60:0/3881251502 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f18f40762d0 0x7f18f4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.588+0000 7f19057fa640 1 -- 172.21.15.60:0/3881251502 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7f18fc096960 con 0x7f190809a610 2024-05-09T05:02:07.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.589+0000 7f190cbf0640 1 --2- 172.21.15.60:0/3881251502 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f18f40762d0 0x7f18f4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.589+0000 7f190cbf0640 1 --2- 172.21.15.60:0/3881251502 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f18f40762d0 0x7f18f4078790 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f18f8034000 tx=0x7f18f802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:07.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.593+0000 7f19057fa640 1 -- 172.21.15.60:0/3881251502 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f18fc05fe70 con 0x7f190809a610 2024-05-09T05:02:07.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.782+0000 7f190ee54640 1 -- 172.21.15.60:0/3881251502 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f1908097ae0 con 0x7f18f40762d0 2024-05-09T05:02:07.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.783+0000 7f19057fa640 1 -- 172.21.15.60:0/3881251502 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1908097ae0 con 0x7f18f40762d0 2024-05-09T05:02:07.783 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 4 to deep-scrub 2024-05-09T05:02:07.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.785+0000 7f190ee54640 1 -- 172.21.15.60:0/3881251502 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f18f40762d0 msgr2=0x7f18f4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.785+0000 7f190ee54640 1 --2- 172.21.15.60:0/3881251502 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f18f40762d0 0x7f18f4078790 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f18f8034000 tx=0x7f18f802e000 comp rx=0 tx=0).stop 2024-05-09T05:02:07.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.785+0000 7f190ee54640 1 -- 172.21.15.60:0/3881251502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f190809a610 msgr2=0x7f19081f2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.785+0000 7f190ee54640 1 --2- 172.21.15.60:0/3881251502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f190809a610 0x7f19081f2d20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f18fc00c770 tx=0x7f18fc00cc30 comp rx=0 tx=0).stop 2024-05-09T05:02:07.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.785+0000 7f190ee54640 1 -- 172.21.15.60:0/3881251502 shutdown_connections 2024-05-09T05:02:07.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.785+0000 7f190ee54640 1 --2- 172.21.15.60:0/3881251502 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f18f40762d0 0x7f18f4078790 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.785+0000 7f190ee54640 1 --2- 172.21.15.60:0/3881251502 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f190815b990 0x7f19081e4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.785+0000 7f190ee54640 1 --2- 172.21.15.60:0/3881251502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f190809a610 0x7f19081f2d20 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.785+0000 7f190ee54640 1 --2- 172.21.15.60:0/3881251502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1908097de0 0x7f19080b1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.785+0000 7f190ee54640 1 -- 172.21.15.60:0/3881251502 >> 172.21.15.60:0/3881251502 conn(0x7f19080972f0 msgr2=0x7f19080ab560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:07.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.785+0000 7f190ee54640 1 -- 172.21.15.60:0/3881251502 shutdown_connections 2024-05-09T05:02:07.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.785+0000 7f190ee54640 1 -- 172.21.15.60:0/3881251502 wait complete. 2024-05-09T05:02:07.797 INFO:tasks.ceph:Scrubbing osd.5 2024-05-09T05:02:07.797 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-05-09T05:02:07.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.904+0000 7f6d96e1c640 1 Processor -- start 2024-05-09T05:02:07.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.904+0000 7f6d96e1c640 1 -- start start 2024-05-09T05:02:07.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.905+0000 7f6d96e1c640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6d90162e70 0x7f6d90163300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.905+0000 7f6d96e1c640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6d900b5e50 0x7f6d900b6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.905+0000 7f6d96e1c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d900d31b0 con 0x7f6d90148120 2024-05-09T05:02:07.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.905+0000 7f6d96e1c640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f6d900d3490 con 0x7f6d900b5e50 2024-05-09T05:02:07.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.905+0000 7f6d96e1c640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f6d900b5650 con 0x7f6d90162e70 2024-05-09T05:02:07.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.905+0000 7f6d953b9640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6d900b5e50 0x7f6d900b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.905+0000 7f6d94bb8640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6d90148120 0x7f6d900d2760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34302/0 (socket says 172.21.15.60:34302) 2024-05-09T05:02:07.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.905+0000 7f6d94bb8640 1 -- 172.21.15.60:0/1829039440 learned_addr learned my addr 172.21.15.60:0/1829039440 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:02:07.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.906+0000 7f6d953b9640 1 -- 172.21.15.60:0/1829039440 >> v2:172.21.15.114:3300/0 conn(0x7f6d90162e70 msgr2=0x7f6d90163300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:07.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.906+0000 7f6d8ffff640 1 --2- 172.21.15.60:0/1829039440 >> v2:172.21.15.114:3300/0 conn(0x7f6d90162e70 0x7f6d90163300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.906+0000 7f6d953b9640 1 --2- 172.21.15.60:0/1829039440 >> v2:172.21.15.114:3300/0 conn(0x7f6d90162e70 0x7f6d90163300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.906+0000 7f6d953b9640 1 -- 172.21.15.60:0/1829039440 >> v1:172.21.15.60:6789/0 conn(0x7f6d90148120 legacy=0x7f6d900d2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.906+0000 7f6d953b9640 1 -- 172.21.15.60:0/1829039440 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d900b5930 con 0x7f6d900b5e50 2024-05-09T05:02:07.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.906+0000 7f6d8f7fe640 1 -- 172.21.15.60:0/1829039440 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2175329845 0 0) 0x7f6d900d31b0 con 0x7f6d90148120 2024-05-09T05:02:07.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.906+0000 7f6d8ffff640 1 --2- 172.21.15.60:0/1829039440 >> v2:172.21.15.114:3300/0 conn(0x7f6d90162e70 0x7f6d90163300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:02:07.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.907+0000 7f6d953b9640 1 --2- 172.21.15.60:0/1829039440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d900b5e50 0x7f6d900b6230 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6d900b4ac0 tx=0x7f6d8802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ea8a436e8cdc32 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:07.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.907+0000 7f6d8f7fe640 1 -- 172.21.15.60:0/1829039440 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d88037070 con 0x7f6d900b5e50 2024-05-09T05:02:07.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.907+0000 7f6d8f7fe640 1 -- 172.21.15.60:0/1829039440 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d88032440 con 0x7f6d900b5e50 2024-05-09T05:02:07.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.907+0000 7f6d8f7fe640 1 -- 172.21.15.60:0/1829039440 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d88032810 con 0x7f6d900b5e50 2024-05-09T05:02:07.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.907+0000 7f6d96e1c640 1 -- 172.21.15.60:0/1829039440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d900b5e50 msgr2=0x7f6d900b6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.907+0000 7f6d96e1c640 1 --2- 172.21.15.60:0/1829039440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d900b5e50 0x7f6d900b6230 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6d900b4ac0 tx=0x7f6d8802a1a0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.907+0000 7f6d96e1c640 1 -- 172.21.15.60:0/1829039440 shutdown_connections 2024-05-09T05:02:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.907+0000 7f6d96e1c640 1 --2- 172.21.15.60:0/1829039440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d900b5e50 0x7f6d900b6230 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.907+0000 7f6d96e1c640 1 --2- 172.21.15.60:0/1829039440 >> v2:172.21.15.114:3300/0 conn(0x7f6d90162e70 0x7f6d90163300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.907+0000 7f6d96e1c640 1 -- 172.21.15.60:0/1829039440 >> 172.21.15.60:0/1829039440 conn(0x7f6d9006f640 msgr2=0x7f6d90073840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.907+0000 7f6d96e1c640 1 -- 172.21.15.60:0/1829039440 shutdown_connections 2024-05-09T05:02:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.907+0000 7f6d96e1c640 1 -- 172.21.15.60:0/1829039440 wait complete. 2024-05-09T05:02:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.908+0000 7f6d96e1c640 1 Processor -- start 2024-05-09T05:02:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.909+0000 7f6d96e1c640 1 -- start start 2024-05-09T05:02:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.909+0000 7f6d96e1c640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d900b5e50 0x7f6d9015eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.909+0000 7f6d96e1c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d90148120 0x7f6d9013f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.909+0000 7f6d96e1c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d90162e70 0x7f6d900b4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.909+0000 7f6d96e1c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6d90163b30 con 0x7f6d90148120 2024-05-09T05:02:07.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.909+0000 7f6d96e1c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6d900d2e70 con 0x7f6d90162e70 2024-05-09T05:02:07.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.909+0000 7f6d96e1c640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6d90163cb0 con 0x7f6d900b5e50 2024-05-09T05:02:07.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.910+0000 7f6d8ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d90148120 0x7f6d9013f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.910+0000 7f6d8ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d90148120 0x7f6d9013f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36540/0 (socket says 172.21.15.60:36540) 2024-05-09T05:02:07.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.910+0000 7f6d8ffff640 1 -- 172.21.15.60:0/74181211 learned_addr learned my addr 172.21.15.60:0/74181211 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:07.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.910+0000 7f6d953b9640 1 --2- 172.21.15.60:0/74181211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d90162e70 0x7f6d900b4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.910+0000 7f6d94bb8640 1 --2- 172.21.15.60:0/74181211 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d900b5e50 0x7f6d9015eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.910+0000 7f6d8ffff640 1 -- 172.21.15.60:0/74181211 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d900b5e50 msgr2=0x7f6d9015eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.910+0000 7f6d8ffff640 1 --2- 172.21.15.60:0/74181211 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6d900b5e50 0x7f6d9015eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.910+0000 7f6d8ffff640 1 -- 172.21.15.60:0/74181211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d90162e70 msgr2=0x7f6d900b4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.910+0000 7f6d8ffff640 1 --2- 172.21.15.60:0/74181211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6d90162e70 0x7f6d900b4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.910+0000 7f6d8ffff640 1 -- 172.21.15.60:0/74181211 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d90162590 con 0x7f6d90148120 2024-05-09T05:02:07.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.910+0000 7f6d8ffff640 1 --2- 172.21.15.60:0/74181211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d90148120 0x7f6d9013f6b0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800c820 tx=0x7f6d7800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:07.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.910+0000 7f6d8d7fa640 1 -- 172.21.15.60:0/74181211 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78015070 con 0x7f6d90148120 2024-05-09T05:02:07.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.910+0000 7f6d8d7fa640 1 -- 172.21.15.60:0/74181211 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d78007a30 con 0x7f6d90148120 2024-05-09T05:02:07.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.910+0000 7f6d96e1c640 1 -- 172.21.15.60:0/74181211 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d90161cf0 con 0x7f6d90148120 2024-05-09T05:02:07.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.910+0000 7f6d8d7fa640 1 -- 172.21.15.60:0/74181211 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78007de0 con 0x7f6d90148120 2024-05-09T05:02:07.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.910+0000 7f6d96e1c640 1 -- 172.21.15.60:0/74181211 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d901622b0 con 0x7f6d90148120 2024-05-09T05:02:07.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.911+0000 7f6d96e1c640 1 -- 172.21.15.60:0/74181211 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6d90051040 con 0x7f6d90148120 2024-05-09T05:02:07.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.912+0000 7f6d8d7fa640 1 -- 172.21.15.60:0/74181211 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6d78018070 con 0x7f6d90148120 2024-05-09T05:02:07.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.912+0000 7f6d8d7fa640 1 --2- 172.21.15.60:0/74181211 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6d700762d0 0x7f6d70078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.912+0000 7f6d8d7fa640 1 -- 172.21.15.60:0/74181211 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7f6d78097340 con 0x7f6d90148120 2024-05-09T05:02:07.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.912+0000 7f6d94bb8640 1 --2- 172.21.15.60:0/74181211 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6d700762d0 0x7f6d70078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.913+0000 7f6d8d7fa640 1 --2- 172.21.15.60:0/74181211 >> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] conn(0x7f6d7007c770 0x7f6d7007ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:07.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.913+0000 7f6d8d7fa640 1 -- 172.21.15.60:0/74181211 --> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d7007f280 con 0x7f6d7007c770 2024-05-09T05:02:07.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.913+0000 7f6d8d7fa640 1 -- 172.21.15.60:0/74181211 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f6d7809e050 con 0x7f6d90148120 2024-05-09T05:02:07.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.913+0000 7f6d953b9640 1 --2- 172.21.15.60:0/74181211 >> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] conn(0x7f6d7007c770 0x7f6d7007ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:07.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.913+0000 7f6d94bb8640 1 --2- 172.21.15.60:0/74181211 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6d700762d0 0x7f6d70078790 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6d800012e0 tx=0x7f6d800016f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:07.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.913+0000 7f6d953b9640 1 --2- 172.21.15.60:0/74181211 >> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] conn(0x7f6d7007c770 0x7f6d7007ebb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:07.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.914+0000 7f6d8d7fa640 1 -- 172.21.15.60:0/74181211 <== osd.5 v2:172.21.15.114:6816/292113074 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6d7007f280 con 0x7f6d7007c770 2024-05-09T05:02:07.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.927+0000 7f6d96e1c640 1 -- 172.21.15.60:0/74181211 --> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6d900d1600 con 0x7f6d7007c770 2024-05-09T05:02:07.928 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.928+0000 7f6d8d7fa640 1 -- 172.21.15.60:0/74181211 <== osd.5 v2:172.21.15.114:6816/292113074 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f6d900d1600 con 0x7f6d7007c770 2024-05-09T05:02:07.928 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:02:07.929 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T05:02:07.929 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:02:07.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.928+0000 7f6d96e1c640 1 -- 172.21.15.60:0/74181211 >> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] conn(0x7f6d7007c770 msgr2=0x7f6d7007ebb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.928+0000 7f6d96e1c640 1 --2- 172.21.15.60:0/74181211 >> [v2:172.21.15.114:6816/292113074,v1:172.21.15.114:6817/292113074] conn(0x7f6d7007c770 0x7f6d7007ebb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.928+0000 7f6d96e1c640 1 -- 172.21.15.60:0/74181211 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6d700762d0 msgr2=0x7f6d70078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.928+0000 7f6d96e1c640 1 --2- 172.21.15.60:0/74181211 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6d700762d0 0x7f6d70078790 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6d800012e0 tx=0x7f6d800016f0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.928+0000 7f6d96e1c640 1 -- 172.21.15.60:0/74181211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d90148120 msgr2=0x7f6d9013f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.928+0000 7f6d96e1c640 1 --2- 172.21.15.60:0/74181211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d90148120 0x7f6d9013f6b0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800c820 tx=0x7f6d7800cce0 comp rx=0 tx=0).stop 2024-05-09T05:02:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.928+0000 7f6d953b9640 1 -- 172.21.15.60:0/74181211 reap_dead start 2024-05-09T05:02:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.928+0000 7f6d96e1c640 1 -- 172.21.15.60:0/74181211 shutdown_connections 2024-05-09T05:02:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.928+0000 7f6d96e1c640 1 -- 172.21.15.60:0/74181211 >> 172.21.15.60:0/74181211 conn(0x7f6d9006f640 msgr2=0x7f6d900b6230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.928+0000 7f6d96e1c640 1 -- 172.21.15.60:0/74181211 shutdown_connections 2024-05-09T05:02:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:07.928+0000 7f6d96e1c640 1 -- 172.21.15.60:0/74181211 wait complete. 2024-05-09T05:02:07.940 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-05-09T05:02:08.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.045+0000 7f6fb8bb6640 1 Processor -- start 2024-05-09T05:02:08.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.045+0000 7f6fb8bb6640 1 -- start start 2024-05-09T05:02:08.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.045+0000 7f6fb8bb6640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6fb406f9f0 0x7f6fb406ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.045+0000 7f6fb8bb6640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fb406ff60 con 0x7f6fb41606f0 2024-05-09T05:02:08.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.045+0000 7f6fb8bb6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fb4070240 con 0x7f6fb415f300 2024-05-09T05:02:08.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.045+0000 7f6fb8bb6640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f6fb4070520 con 0x7f6fb406f9f0 2024-05-09T05:02:08.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.046+0000 7f6fb2d9d640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6fb406f9f0 0x7f6fb406ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.046+0000 7f6fabfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6fb41606f0 0x7f6fb40d33f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34306/0 (socket says 172.21.15.60:34306) 2024-05-09T05:02:08.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.046+0000 7f6fabfff640 1 -- 172.21.15.60:0/920234631 learned_addr learned my addr 172.21.15.60:0/920234631 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:02:08.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.046+0000 7f6fb2d9d640 1 -- 172.21.15.60:0/920234631 >> v1:172.21.15.114:6789/0 conn(0x7f6fb415f300 legacy=0x7f6fb41639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:08.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.046+0000 7f6fb2d9d640 1 -- 172.21.15.60:0/920234631 >> v1:172.21.15.60:6789/0 conn(0x7f6fb41606f0 legacy=0x7f6fb40d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.046+0000 7f6fb2d9d640 1 -- 172.21.15.60:0/920234631 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fb406e740 con 0x7f6fb406f9f0 2024-05-09T05:02:08.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.046+0000 7f6fb1d9b640 1 -- 172.21.15.60:0/920234631 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4053992896 0 0) 0x7f6fb406ff60 con 0x7f6fb41606f0 2024-05-09T05:02:08.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.047+0000 7f6fb2d9d640 1 --2- 172.21.15.60:0/920234631 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fb406f9f0 0x7f6fb406ef40 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f6fac009dc0 tx=0x7f6fac02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=561adaff21d529a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:08.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.047+0000 7f6fb1d9b640 1 -- 172.21.15.60:0/920234631 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fac02ae60 con 0x7f6fb406f9f0 2024-05-09T05:02:08.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.047+0000 7f6fb1d9b640 1 -- 172.21.15.60:0/920234631 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fac007740 con 0x7f6fb406f9f0 2024-05-09T05:02:08.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.047+0000 7f6fb1d9b640 1 -- 172.21.15.60:0/920234631 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fac007b10 con 0x7f6fb406f9f0 2024-05-09T05:02:08.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.047+0000 7f6fb8bb6640 1 -- 172.21.15.60:0/920234631 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fb406f9f0 msgr2=0x7f6fb406ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.047+0000 7f6fb8bb6640 1 --2- 172.21.15.60:0/920234631 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fb406f9f0 0x7f6fb406ef40 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f6fac009dc0 tx=0x7f6fac02a680 comp rx=0 tx=0).stop 2024-05-09T05:02:08.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.047+0000 7f6fb8bb6640 1 -- 172.21.15.60:0/920234631 shutdown_connections 2024-05-09T05:02:08.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.047+0000 7f6fb8bb6640 1 --2- 172.21.15.60:0/920234631 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fb406f9f0 0x7f6fb406ef40 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.047+0000 7f6fb8bb6640 1 -- 172.21.15.60:0/920234631 >> 172.21.15.60:0/920234631 conn(0x7f6fb409b850 msgr2=0x7f6fb409bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:08.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.047+0000 7f6fb8bb6640 1 -- 172.21.15.60:0/920234631 shutdown_connections 2024-05-09T05:02:08.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.047+0000 7f6fb8bb6640 1 -- 172.21.15.60:0/920234631 wait complete. 2024-05-09T05:02:08.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.049+0000 7f6fb8bb6640 1 Processor -- start 2024-05-09T05:02:08.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.049+0000 7f6fb8bb6640 1 -- start start 2024-05-09T05:02:08.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6fb8bb6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fb406f9f0 0x7f6fb409f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6fb8bb6640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6fb415f300 0x7f6fb40b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6fb8bb6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fb41606f0 0x7f6fb4161c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6fb8bb6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb40d3c80 con 0x7f6fb406f9f0 2024-05-09T05:02:08.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6fb8bb6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb40d3b00 con 0x7f6fb41606f0 2024-05-09T05:02:08.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6fb8bb6640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6fb406f480 con 0x7f6fb415f300 2024-05-09T05:02:08.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6fb259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fb406f9f0 0x7f6fb409f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6fb259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fb406f9f0 0x7f6fb409f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36550/0 (socket says 172.21.15.60:36550) 2024-05-09T05:02:08.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6fb259c640 1 -- 172.21.15.60:0/2734408117 learned_addr learned my addr 172.21.15.60:0/2734408117 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:08.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6fabfff640 1 --2- 172.21.15.60:0/2734408117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6fb415f300 0x7f6fb40b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6fb2d9d640 1 --2- 172.21.15.60:0/2734408117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fb41606f0 0x7f6fb4161c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6fb259c640 1 -- 172.21.15.60:0/2734408117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6fb415f300 msgr2=0x7f6fb40b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6fb259c640 1 --2- 172.21.15.60:0/2734408117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6fb415f300 0x7f6fb40b3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6fb259c640 1 -- 172.21.15.60:0/2734408117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fb41606f0 msgr2=0x7f6fb4161c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6fb259c640 1 --2- 172.21.15.60:0/2734408117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fb41606f0 0x7f6fb4161c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6fb259c640 1 -- 172.21.15.60:0/2734408117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fb40b4250 con 0x7f6fb406f9f0 2024-05-09T05:02:08.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6fb2d9d640 1 --2- 172.21.15.60:0/2734408117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fb41606f0 0x7f6fb4161c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:02:08.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6fb259c640 1 --2- 172.21.15.60:0/2734408117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fb406f9f0 0x7f6fb409f2c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6fa400c430 tx=0x7f6fa400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:08.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6faaffd640 1 -- 172.21.15.60:0/2734408117 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa4011790 con 0x7f6fb406f9f0 2024-05-09T05:02:08.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6faaffd640 1 -- 172.21.15.60:0/2734408117 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fa4017070 con 0x7f6fb406f9f0 2024-05-09T05:02:08.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6fb8bb6640 1 -- 172.21.15.60:0/2734408117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fb40b2c70 con 0x7f6fb406f9f0 2024-05-09T05:02:08.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.050+0000 7f6faaffd640 1 -- 172.21.15.60:0/2734408117 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa4011bb0 con 0x7f6fb406f9f0 2024-05-09T05:02:08.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.051+0000 7f6fb8bb6640 1 -- 172.21.15.60:0/2734408117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fb40b3230 con 0x7f6fb406f9f0 2024-05-09T05:02:08.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.051+0000 7f6fb8bb6640 1 -- 172.21.15.60:0/2734408117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fb4072020 con 0x7f6fb406f9f0 2024-05-09T05:02:08.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.052+0000 7f6faaffd640 1 -- 172.21.15.60:0/2734408117 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6fa401c460 con 0x7f6fb406f9f0 2024-05-09T05:02:08.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.053+0000 7f6faaffd640 1 --2- 172.21.15.60:0/2734408117 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6f94076300 0x7f6f940787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.053+0000 7f6faaffd640 1 -- 172.21.15.60:0/2734408117 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7f6fa40972b0 con 0x7f6fb406f9f0 2024-05-09T05:02:08.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.053+0000 7f6fabfff640 1 --2- 172.21.15.60:0/2734408117 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6f94076300 0x7f6f940787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.053+0000 7f6fabfff640 1 --2- 172.21.15.60:0/2734408117 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6f94076300 0x7f6f940787c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6fb41f8950 tx=0x7f6f9c007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:08.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.057+0000 7f6faaffd640 1 -- 172.21.15.60:0/2734408117 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6fb4072020 con 0x7f6fb406f9f0 2024-05-09T05:02:08.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.247+0000 7f6fb8bb6640 1 -- 172.21.15.60:0/2734408117 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f6fb40b3b60 con 0x7f6f94076300 2024-05-09T05:02:08.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.249+0000 7f6faaffd640 1 -- 172.21.15.60:0/2734408117 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6fb40b3b60 con 0x7f6f94076300 2024-05-09T05:02:08.249 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 5 to deep-scrub 2024-05-09T05:02:08.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.250+0000 7f6fb8bb6640 1 -- 172.21.15.60:0/2734408117 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6f94076300 msgr2=0x7f6f940787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.250+0000 7f6fb8bb6640 1 --2- 172.21.15.60:0/2734408117 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6f94076300 0x7f6f940787c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6fb41f8950 tx=0x7f6f9c007de0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.250+0000 7f6fb8bb6640 1 -- 172.21.15.60:0/2734408117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fb406f9f0 msgr2=0x7f6fb409f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.250+0000 7f6fb8bb6640 1 --2- 172.21.15.60:0/2734408117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fb406f9f0 0x7f6fb409f2c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6fa400c430 tx=0x7f6fa400c8f0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.250+0000 7f6fb8bb6640 1 -- 172.21.15.60:0/2734408117 shutdown_connections 2024-05-09T05:02:08.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.250+0000 7f6fb8bb6640 1 --2- 172.21.15.60:0/2734408117 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6f94076300 0x7f6f940787c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.250+0000 7f6fb8bb6640 1 --2- 172.21.15.60:0/2734408117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fb41606f0 0x7f6fb4161c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.250+0000 7f6fb8bb6640 1 --2- 172.21.15.60:0/2734408117 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6fb415f300 0x7f6fb40b3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.250+0000 7f6fb8bb6640 1 --2- 172.21.15.60:0/2734408117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6fb406f9f0 0x7f6fb409f2c0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.250+0000 7f6fb8bb6640 1 -- 172.21.15.60:0/2734408117 >> 172.21.15.60:0/2734408117 conn(0x7f6fb409b850 msgr2=0x7f6fb40d2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:08.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.250+0000 7f6fb8bb6640 1 -- 172.21.15.60:0/2734408117 shutdown_connections 2024-05-09T05:02:08.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.250+0000 7f6fb8bb6640 1 -- 172.21.15.60:0/2734408117 wait complete. 2024-05-09T05:02:08.261 INFO:tasks.ceph:Scrubbing osd.6 2024-05-09T05:02:08.262 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-05-09T05:02:08.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.369+0000 7f8a6dc74640 1 Processor -- start 2024-05-09T05:02:08.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.369+0000 7f8a6dc74640 1 -- start start 2024-05-09T05:02:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.370+0000 7f8a6dc74640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f8a680ba8b0 0x7f8a680b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.370+0000 7f8a6dc74640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a680b84e0 0x7f8a680d2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.370+0000 7f8a6dc74640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a680b5a50 con 0x7f8a6815f260 2024-05-09T05:02:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.370+0000 7f8a6dc74640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f8a680b5d30 con 0x7f8a680ba8b0 2024-05-09T05:02:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.370+0000 7f8a6dc74640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8a680b6010 con 0x7f8a680b84e0 2024-05-09T05:02:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.370+0000 7f8a677fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8a6815f260 0x7f8a680ba1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34308/0 (socket says 172.21.15.60:34308) 2024-05-09T05:02:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.370+0000 7f8a677fe640 1 -- 172.21.15.60:0/2056874357 learned_addr learned my addr 172.21.15.60:0/2056874357 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:02:08.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.370+0000 7f8a67fff640 1 --2- 172.21.15.60:0/2056874357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a680b84e0 0x7f8a680d2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.370+0000 7f8a66ffd640 1 --2- 172.21.15.60:0/2056874357 >> v2:172.21.15.114:3300/0 conn(0x7f8a680ba8b0 0x7f8a680b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.370+0000 7f8a667fc640 1 -- 172.21.15.60:0/2056874357 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3501973581 0 0) 0x7f8a680b5a50 con 0x7f8a6815f260 2024-05-09T05:02:08.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.370+0000 7f8a667fc640 1 -- 172.21.15.60:0/2056874357 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a50003520 con 0x7f8a6815f260 2024-05-09T05:02:08.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.370+0000 7f8a67fff640 1 -- 172.21.15.60:0/2056874357 >> v2:172.21.15.114:3300/0 conn(0x7f8a680ba8b0 msgr2=0x7f8a680b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:08.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.370+0000 7f8a67fff640 1 --2- 172.21.15.60:0/2056874357 >> v2:172.21.15.114:3300/0 conn(0x7f8a680ba8b0 0x7f8a680b6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.371+0000 7f8a67fff640 1 -- 172.21.15.60:0/2056874357 >> v1:172.21.15.60:6789/0 conn(0x7f8a6815f260 legacy=0x7f8a680ba1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.371+0000 7f8a67fff640 1 -- 172.21.15.60:0/2056874357 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a680b62f0 con 0x7f8a680b84e0 2024-05-09T05:02:08.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.371+0000 7f8a66ffd640 1 --2- 172.21.15.60:0/2056874357 >> v2:172.21.15.114:3300/0 conn(0x7f8a680ba8b0 0x7f8a680b6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:02:08.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.371+0000 7f8a67fff640 1 --2- 172.21.15.60:0/2056874357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a680b84e0 0x7f8a680d2af0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009d90 tx=0x7f8a6002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c45b55b17f50013 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:08.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.371+0000 7f8a667fc640 1 -- 172.21.15.60:0/2056874357 <== mon.2 v2:172.21.15.114:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a60007740 con 0x7f8a680b84e0 2024-05-09T05:02:08.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.372+0000 7f8a667fc640 1 -- 172.21.15.60:0/2056874357 <== mon.2 v2:172.21.15.114:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a600078e0 con 0x7f8a680b84e0 2024-05-09T05:02:08.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.372+0000 7f8a667fc640 1 -- 172.21.15.60:0/2056874357 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a60007cb0 con 0x7f8a680b84e0 2024-05-09T05:02:08.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.372+0000 7f8a6dc74640 1 -- 172.21.15.60:0/2056874357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a680b84e0 msgr2=0x7f8a680d2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.372+0000 7f8a6dc74640 1 --2- 172.21.15.60:0/2056874357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a680b84e0 0x7f8a680d2af0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009d90 tx=0x7f8a6002a660 comp rx=0 tx=0).stop 2024-05-09T05:02:08.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.372+0000 7f8a6dc74640 1 -- 172.21.15.60:0/2056874357 shutdown_connections 2024-05-09T05:02:08.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.372+0000 7f8a6dc74640 1 --2- 172.21.15.60:0/2056874357 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a680b84e0 0x7f8a680d2af0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.372+0000 7f8a6dc74640 1 --2- 172.21.15.60:0/2056874357 >> v2:172.21.15.114:3300/0 conn(0x7f8a680ba8b0 0x7f8a680b6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.372+0000 7f8a6dc74640 1 -- 172.21.15.60:0/2056874357 >> 172.21.15.60:0/2056874357 conn(0x7f8a6806ede0 msgr2=0x7f8a68073820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:08.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.372+0000 7f8a6dc74640 1 -- 172.21.15.60:0/2056874357 shutdown_connections 2024-05-09T05:02:08.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.372+0000 7f8a6dc74640 1 -- 172.21.15.60:0/2056874357 wait complete. 2024-05-09T05:02:08.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.373+0000 7f8a6dc74640 1 Processor -- start 2024-05-09T05:02:08.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.373+0000 7f8a6dc74640 1 -- start start 2024-05-09T05:02:08.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.374+0000 7f8a6dc74640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a680b84e0 0x7f8a6814aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.374+0000 7f8a6dc74640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a680ba8b0 0x7f8a680b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.374+0000 7f8a6dc74640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a6815f260 0x7f8a6815d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.374+0000 7f8a6dc74640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6815f640 con 0x7f8a680b84e0 2024-05-09T05:02:08.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.374+0000 7f8a6dc74640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8a680b7920 con 0x7f8a680ba8b0 2024-05-09T05:02:08.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.374+0000 7f8a6dc74640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f8a6815f7c0 con 0x7f8a6815f260 2024-05-09T05:02:08.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.374+0000 7f8a677fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a680b84e0 0x7f8a6814aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.374+0000 7f8a677fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a680b84e0 0x7f8a6814aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36562/0 (socket says 172.21.15.60:36562) 2024-05-09T05:02:08.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.374+0000 7f8a677fe640 1 -- 172.21.15.60:0/1936186223 learned_addr learned my addr 172.21.15.60:0/1936186223 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:08.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.374+0000 7f8a66ffd640 1 --2- 172.21.15.60:0/1936186223 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a680ba8b0 0x7f8a680b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.374+0000 7f8a67fff640 1 --2- 172.21.15.60:0/1936186223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a6815f260 0x7f8a6815d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.375+0000 7f8a677fe640 1 -- 172.21.15.60:0/1936186223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a6815f260 msgr2=0x7f8a6815d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.375+0000 7f8a677fe640 1 --2- 172.21.15.60:0/1936186223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a6815f260 0x7f8a6815d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.375+0000 7f8a677fe640 1 -- 172.21.15.60:0/1936186223 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a680ba8b0 msgr2=0x7f8a680b0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.375+0000 7f8a677fe640 1 --2- 172.21.15.60:0/1936186223 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a680ba8b0 0x7f8a680b0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.375+0000 7f8a677fe640 1 -- 172.21.15.60:0/1936186223 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6814b200 con 0x7f8a680b84e0 2024-05-09T05:02:08.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.375+0000 7f8a67fff640 1 --2- 172.21.15.60:0/1936186223 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f8a6815f260 0x7f8a6815d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:02:08.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.375+0000 7f8a677fe640 1 --2- 172.21.15.60:0/1936186223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a680b84e0 0x7f8a6814aad0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8a5800c400 tx=0x7f8a5800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:08.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.375+0000 7f8a4ffff640 1 -- 172.21.15.60:0/1936186223 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a58017070 con 0x7f8a680b84e0 2024-05-09T05:02:08.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.375+0000 7f8a4ffff640 1 -- 172.21.15.60:0/1936186223 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a58005ce0 con 0x7f8a680b84e0 2024-05-09T05:02:08.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.375+0000 7f8a6dc74640 1 -- 172.21.15.60:0/1936186223 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a6814d800 con 0x7f8a680b84e0 2024-05-09T05:02:08.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.375+0000 7f8a4ffff640 1 -- 172.21.15.60:0/1936186223 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a580072d0 con 0x7f8a680b84e0 2024-05-09T05:02:08.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.375+0000 7f8a6dc74640 1 -- 172.21.15.60:0/1936186223 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a6814ddc0 con 0x7f8a680b84e0 2024-05-09T05:02:08.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.376+0000 7f8a6dc74640 1 -- 172.21.15.60:0/1936186223 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8a68051040 con 0x7f8a680b84e0 2024-05-09T05:02:08.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.377+0000 7f8a4ffff640 1 -- 172.21.15.60:0/1936186223 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8a5801a070 con 0x7f8a680b84e0 2024-05-09T05:02:08.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.377+0000 7f8a4ffff640 1 --2- 172.21.15.60:0/1936186223 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8a54076300 0x7f8a540787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.377+0000 7f8a4ffff640 1 -- 172.21.15.60:0/1936186223 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7f8a580966e0 con 0x7f8a680b84e0 2024-05-09T05:02:08.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.377+0000 7f8a4ffff640 1 --2- 172.21.15.60:0/1936186223 >> [v2:172.21.15.114:6824/2612522523,v1:172.21.15.114:6825/2612522523] conn(0x7f8a5407c7a0 0x7f8a5407ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.377+0000 7f8a4ffff640 1 -- 172.21.15.60:0/1936186223 --> [v2:172.21.15.114:6824/2612522523,v1:172.21.15.114:6825/2612522523] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8a5407f2b0 con 0x7f8a5407c7a0 2024-05-09T05:02:08.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.377+0000 7f8a4ffff640 1 -- 172.21.15.60:0/1936186223 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f8a5809d050 con 0x7f8a680b84e0 2024-05-09T05:02:08.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.377+0000 7f8a66ffd640 1 --2- 172.21.15.60:0/1936186223 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8a54076300 0x7f8a540787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.378+0000 7f8a67fff640 1 --2- 172.21.15.60:0/1936186223 >> [v2:172.21.15.114:6824/2612522523,v1:172.21.15.114:6825/2612522523] conn(0x7f8a5407c7a0 0x7f8a5407ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.378+0000 7f8a66ffd640 1 --2- 172.21.15.60:0/1936186223 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8a54076300 0x7f8a540787c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c009560 tx=0x7f8a5c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:08.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.378+0000 7f8a67fff640 1 --2- 172.21.15.60:0/1936186223 >> [v2:172.21.15.114:6824/2612522523,v1:172.21.15.114:6825/2612522523] conn(0x7f8a5407c7a0 0x7f8a5407ebe0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:08.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.379+0000 7f8a4ffff640 1 -- 172.21.15.60:0/1936186223 <== osd.6 v2:172.21.15.114:6824/2612522523 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8a5407f2b0 con 0x7f8a5407c7a0 2024-05-09T05:02:08.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.392+0000 7f8a6dc74640 1 -- 172.21.15.60:0/1936186223 --> [v2:172.21.15.114:6824/2612522523,v1:172.21.15.114:6825/2612522523] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8a680d4760 con 0x7f8a5407c7a0 2024-05-09T05:02:08.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.393+0000 7f8a4ffff640 1 -- 172.21.15.60:0/1936186223 <== osd.6 v2:172.21.15.114:6824/2612522523 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f8a680d4760 con 0x7f8a5407c7a0 2024-05-09T05:02:08.393 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:02:08.394 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T05:02:08.394 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:02:08.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.393+0000 7f8a6dc74640 1 -- 172.21.15.60:0/1936186223 >> [v2:172.21.15.114:6824/2612522523,v1:172.21.15.114:6825/2612522523] conn(0x7f8a5407c7a0 msgr2=0x7f8a5407ebe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.393+0000 7f8a6dc74640 1 --2- 172.21.15.60:0/1936186223 >> [v2:172.21.15.114:6824/2612522523,v1:172.21.15.114:6825/2612522523] conn(0x7f8a5407c7a0 0x7f8a5407ebe0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.393+0000 7f8a6dc74640 1 -- 172.21.15.60:0/1936186223 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8a54076300 msgr2=0x7f8a540787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.393+0000 7f8a6dc74640 1 --2- 172.21.15.60:0/1936186223 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f8a54076300 0x7f8a540787c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c009560 tx=0x7f8a5c009190 comp rx=0 tx=0).stop 2024-05-09T05:02:08.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.393+0000 7f8a6dc74640 1 -- 172.21.15.60:0/1936186223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a680b84e0 msgr2=0x7f8a6814aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.393+0000 7f8a6dc74640 1 --2- 172.21.15.60:0/1936186223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a680b84e0 0x7f8a6814aad0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8a5800c400 tx=0x7f8a5800c8c0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.393+0000 7f8a67fff640 1 -- 172.21.15.60:0/1936186223 reap_dead start 2024-05-09T05:02:08.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.393+0000 7f8a6dc74640 1 -- 172.21.15.60:0/1936186223 shutdown_connections 2024-05-09T05:02:08.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.393+0000 7f8a6dc74640 1 -- 172.21.15.60:0/1936186223 >> 172.21.15.60:0/1936186223 conn(0x7f8a6806ede0 msgr2=0x7f8a681622f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:08.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.393+0000 7f8a6dc74640 1 -- 172.21.15.60:0/1936186223 shutdown_connections 2024-05-09T05:02:08.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.393+0000 7f8a6dc74640 1 -- 172.21.15.60:0/1936186223 wait complete. 2024-05-09T05:02:08.404 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-05-09T05:02:08.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.508+0000 7f1ab00c4640 1 Processor -- start 2024-05-09T05:02:08.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.508+0000 7f1ab00c4640 1 -- start start 2024-05-09T05:02:08.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.509+0000 7f1ab00c4640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f1aa8073fb0 0x7f1aa80b4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.509+0000 7f1ab00c4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1aa815f7a0 con 0x7f1aa815f3c0 2024-05-09T05:02:08.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.509+0000 7f1ab00c4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1aa809f480 con 0x7f1aa80b4b60 2024-05-09T05:02:08.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.509+0000 7f1ab00c4640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f1aa8071cb0 con 0x7f1aa8073fb0 2024-05-09T05:02:08.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.509+0000 7f1aade60640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1aa815f3c0 0x7f1aa809ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34312/0 (socket says 172.21.15.60:34312) 2024-05-09T05:02:08.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.509+0000 7f1aade60640 1 -- 172.21.15.60:0/177167327 learned_addr learned my addr 172.21.15.60:0/177167327 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:02:08.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.509+0000 7f1aad65f640 1 --2- 172.21.15.60:0/177167327 >> v2:172.21.15.114:3300/0 conn(0x7f1aa8073fb0 0x7f1aa80b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.509+0000 7f1aace5e640 1 -- 172.21.15.60:0/177167327 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3199871226 0 0) 0x7f1aa815f7a0 con 0x7f1aa815f3c0 2024-05-09T05:02:08.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.509+0000 7f1aace5e640 1 -- 172.21.15.60:0/177167327 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a94003520 con 0x7f1aa815f3c0 2024-05-09T05:02:08.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aace5e640 1 -- 172.21.15.60:0/177167327 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 828986904 0 0) 0x7f1a94003520 con 0x7f1aa815f3c0 2024-05-09T05:02:08.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aace5e640 1 -- 172.21.15.60:0/177167327 >> v2:172.21.15.114:3300/0 conn(0x7f1aa8073fb0 msgr2=0x7f1aa80b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:08.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aace5e640 1 --2- 172.21.15.60:0/177167327 >> v2:172.21.15.114:3300/0 conn(0x7f1aa8073fb0 0x7f1aa80b4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aace5e640 1 -- 172.21.15.60:0/177167327 >> v1:172.21.15.114:6789/0 conn(0x7f1aa80b4b60 legacy=0x7f1aa8072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aace5e640 1 -- 172.21.15.60:0/177167327 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa809f760 con 0x7f1aa815f3c0 2024-05-09T05:02:08.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aace5e640 1 -- 172.21.15.60:0/177167327 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f1aa40016f0 con 0x7f1aa815f3c0 2024-05-09T05:02:08.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aace5e640 1 -- 172.21.15.60:0/177167327 >> v1:172.21.15.60:6789/0 conn(0x7f1aa815f3c0 legacy=0x7f1aa809ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aace5e640 1 --2- 172.21.15.60:0/177167327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a94003a50 0x7f1a94003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aace5e640 1 --2- 172.21.15.60:0/177167327 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a94004800 0x7f1a94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aace5e640 1 --2- 172.21.15.60:0/177167327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a940052e0 0x7f1a9400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aace5e640 1 -- 172.21.15.60:0/177167327 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa8071cb0 con 0x7f1a94003a50 2024-05-09T05:02:08.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aace5e640 1 -- 172.21.15.60:0/177167327 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9400c1b0 con 0x7f1a940052e0 2024-05-09T05:02:08.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aace5e640 1 -- 172.21.15.60:0/177167327 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1a9400c3b0 con 0x7f1a94004800 2024-05-09T05:02:08.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aade60640 1 --2- 172.21.15.60:0/177167327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a94003a50 0x7f1a94003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aad65f640 1 --2- 172.21.15.60:0/177167327 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a94004800 0x7f1a94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aae661640 1 --2- 172.21.15.60:0/177167327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a940052e0 0x7f1a9400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aade60640 1 -- 172.21.15.60:0/177167327 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a94004800 msgr2=0x7f1a94004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:08.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aade60640 1 --2- 172.21.15.60:0/177167327 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a94004800 0x7f1a94004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aade60640 1 -- 172.21.15.60:0/177167327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a940052e0 msgr2=0x7f1a9400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:08.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aade60640 1 --2- 172.21.15.60:0/177167327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a940052e0 0x7f1a9400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aade60640 1 -- 172.21.15.60:0/177167327 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a94003520 con 0x7f1a94003a50 2024-05-09T05:02:08.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aae661640 1 -- 172.21.15.60:0/177167327 reap_dead start 2024-05-09T05:02:08.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.510+0000 7f1aad65f640 1 --2- 172.21.15.60:0/177167327 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a94004800 0x7f1a94004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:02:08.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.511+0000 7f1aade60640 1 --2- 172.21.15.60:0/177167327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a94003a50 0x7f1a94003e70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f1aa400c210 tx=0x7f1a94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71849d69ace439ae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:08.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.511+0000 7f1aace5e640 1 -- 172.21.15.60:0/177167327 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4007d60 con 0x7f1a94003a50 2024-05-09T05:02:08.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.511+0000 7f1aace5e640 1 -- 172.21.15.60:0/177167327 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aa4005ad0 con 0x7f1a94003a50 2024-05-09T05:02:08.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.511+0000 7f1aace5e640 1 -- 172.21.15.60:0/177167327 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4007070 con 0x7f1a94003a50 2024-05-09T05:02:08.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.511+0000 7f1ab00c4640 1 -- 172.21.15.60:0/177167327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a94003a50 msgr2=0x7f1a94003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.511+0000 7f1ab00c4640 1 --2- 172.21.15.60:0/177167327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a94003a50 0x7f1a94003e70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f1aa400c210 tx=0x7f1a94005830 comp rx=0 tx=0).stop 2024-05-09T05:02:08.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.512+0000 7f1ab00c4640 1 -- 172.21.15.60:0/177167327 shutdown_connections 2024-05-09T05:02:08.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.512+0000 7f1ab00c4640 1 --2- 172.21.15.60:0/177167327 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a94003a50 0x7f1a94003e70 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.512+0000 7f1ab00c4640 1 -- 172.21.15.60:0/177167327 >> 172.21.15.60:0/177167327 conn(0x7f1aa80d1470 msgr2=0x7f1aa80d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:08.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.512+0000 7f1ab00c4640 1 -- 172.21.15.60:0/177167327 shutdown_connections 2024-05-09T05:02:08.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.512+0000 7f1ab00c4640 1 -- 172.21.15.60:0/177167327 wait complete. 2024-05-09T05:02:08.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.513+0000 7f1ab00c4640 1 Processor -- start 2024-05-09T05:02:08.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.513+0000 7f1ab00c4640 1 -- start start 2024-05-09T05:02:08.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.514+0000 7f1ab00c4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a94003a50 0x7f1aa815ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.514+0000 7f1ab00c4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1aa80b3410 0x7f1aa8161a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.514+0000 7f1ab00c4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1aa80b2750 0x7f1aa80b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.514+0000 7f1ab00c4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa80b4620 con 0x7f1aa80b3410 2024-05-09T05:02:08.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.514+0000 7f1ab00c4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa80b4f90 con 0x7f1aa80b2750 2024-05-09T05:02:08.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.514+0000 7f1ab00c4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f1aa80b47a0 con 0x7f1a94003a50 2024-05-09T05:02:08.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.514+0000 7f1aad65f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1aa80b3410 0x7f1aa8161a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.514+0000 7f1aad65f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1aa80b3410 0x7f1aa8161a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36580/0 (socket says 172.21.15.60:36580) 2024-05-09T05:02:08.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.514+0000 7f1aad65f640 1 -- 172.21.15.60:0/906756075 learned_addr learned my addr 172.21.15.60:0/906756075 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:08.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.514+0000 7f1aade60640 1 --2- 172.21.15.60:0/906756075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a94003a50 0x7f1aa815ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.514+0000 7f1aae661640 1 --2- 172.21.15.60:0/906756075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1aa80b2750 0x7f1aa80b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.514+0000 7f1aad65f640 1 -- 172.21.15.60:0/906756075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a94003a50 msgr2=0x7f1aa815ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.514+0000 7f1aad65f640 1 --2- 172.21.15.60:0/906756075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a94003a50 0x7f1aa815ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.514+0000 7f1aad65f640 1 -- 172.21.15.60:0/906756075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1aa80b2750 msgr2=0x7f1aa80b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.514+0000 7f1aad65f640 1 --2- 172.21.15.60:0/906756075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1aa80b2750 0x7f1aa80b2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.514+0000 7f1aad65f640 1 -- 172.21.15.60:0/906756075 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa80cfa60 con 0x7f1aa80b3410 2024-05-09T05:02:08.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.514+0000 7f1aade60640 1 --2- 172.21.15.60:0/906756075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a94003a50 0x7f1aa815ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:02:08.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.514+0000 7f1aae661640 1 --2- 172.21.15.60:0/906756075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1aa80b2750 0x7f1aa80b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:02:08.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.514+0000 7f1aad65f640 1 --2- 172.21.15.60:0/906756075 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1aa80b3410 0x7f1aa8161a30 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1a980114e0 tx=0x7f1a980119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:08.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.515+0000 7f1a9e7fc640 1 -- 172.21.15.60:0/906756075 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a98015870 con 0x7f1aa80b3410 2024-05-09T05:02:08.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.515+0000 7f1a9e7fc640 1 -- 172.21.15.60:0/906756075 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a98015a10 con 0x7f1aa80b3410 2024-05-09T05:02:08.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.515+0000 7f1ab00c4640 1 -- 172.21.15.60:0/906756075 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aa80cfd40 con 0x7f1aa80b3410 2024-05-09T05:02:08.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.515+0000 7f1a9e7fc640 1 -- 172.21.15.60:0/906756075 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a98015e00 con 0x7f1aa80b3410 2024-05-09T05:02:08.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.515+0000 7f1ab00c4640 1 -- 172.21.15.60:0/906756075 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aa81f63f0 con 0x7f1aa80b3410 2024-05-09T05:02:08.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.516+0000 7f1ab00c4640 1 -- 172.21.15.60:0/906756075 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aa80b3170 con 0x7f1aa80b3410 2024-05-09T05:02:08.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.516+0000 7f1a9e7fc640 1 -- 172.21.15.60:0/906756075 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1a9801d400 con 0x7f1aa80b3410 2024-05-09T05:02:08.523 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.517+0000 7f1a9e7fc640 1 --2- 172.21.15.60:0/906756075 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1a8c0762d0 0x7f1a8c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.517+0000 7f1a9e7fc640 1 -- 172.21.15.60:0/906756075 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7f1a98097c60 con 0x7f1aa80b3410 2024-05-09T05:02:08.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.517+0000 7f1aade60640 1 --2- 172.21.15.60:0/906756075 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1a8c0762d0 0x7f1a8c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.518+0000 7f1aade60640 1 --2- 172.21.15.60:0/906756075 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1a8c0762d0 0x7f1a8c078790 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4032000 tx=0x7f1aa4007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:08.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.521+0000 7f1a9e7fc640 1 -- 172.21.15.60:0/906756075 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1a98019030 con 0x7f1aa80b3410 2024-05-09T05:02:08.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.715+0000 7f1ab00c4640 1 -- 172.21.15.60:0/906756075 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f1aa8051040 con 0x7f1a8c0762d0 2024-05-09T05:02:08.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.716+0000 7f1a9e7fc640 1 -- 172.21.15.60:0/906756075 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1aa8051040 con 0x7f1a8c0762d0 2024-05-09T05:02:08.716 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 6 to deep-scrub 2024-05-09T05:02:08.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.718+0000 7f1ab00c4640 1 -- 172.21.15.60:0/906756075 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1a8c0762d0 msgr2=0x7f1a8c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.718+0000 7f1ab00c4640 1 --2- 172.21.15.60:0/906756075 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1a8c0762d0 0x7f1a8c078790 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4032000 tx=0x7f1aa4007000 comp rx=0 tx=0).stop 2024-05-09T05:02:08.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.718+0000 7f1ab00c4640 1 -- 172.21.15.60:0/906756075 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1aa80b3410 msgr2=0x7f1aa8161a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.718+0000 7f1ab00c4640 1 --2- 172.21.15.60:0/906756075 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1aa80b3410 0x7f1aa8161a30 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1a980114e0 tx=0x7f1a980119a0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.718+0000 7f1ab00c4640 1 -- 172.21.15.60:0/906756075 shutdown_connections 2024-05-09T05:02:08.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.718+0000 7f1ab00c4640 1 --2- 172.21.15.60:0/906756075 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f1a8c0762d0 0x7f1a8c078790 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.718+0000 7f1ab00c4640 1 --2- 172.21.15.60:0/906756075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1aa80b2750 0x7f1aa80b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.718+0000 7f1ab00c4640 1 --2- 172.21.15.60:0/906756075 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1aa80b3410 0x7f1aa8161a30 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.718+0000 7f1ab00c4640 1 --2- 172.21.15.60:0/906756075 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f1a94003a50 0x7f1aa815ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.718+0000 7f1ab00c4640 1 -- 172.21.15.60:0/906756075 >> 172.21.15.60:0/906756075 conn(0x7f1aa80d1470 msgr2=0x7f1aa809a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:08.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.718+0000 7f1ab00c4640 1 -- 172.21.15.60:0/906756075 shutdown_connections 2024-05-09T05:02:08.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.718+0000 7f1ab00c4640 1 -- 172.21.15.60:0/906756075 wait complete. 2024-05-09T05:02:08.729 INFO:tasks.ceph:Scrubbing osd.7 2024-05-09T05:02:08.730 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-05-09T05:02:08.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.837+0000 7f3443a09640 1 Processor -- start 2024-05-09T05:02:08.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.837+0000 7f3443a09640 1 -- start start 2024-05-09T05:02:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.838+0000 7f3443a09640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f343c072840 0x7f343c1450f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.838+0000 7f3443a09640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f343c0cf200 con 0x7f343c15f9b0 2024-05-09T05:02:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.838+0000 7f3443a09640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f343c15ffa0 con 0x7f343c162fd0 2024-05-09T05:02:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.838+0000 7f3443a09640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f343c144c60 con 0x7f343c072840 2024-05-09T05:02:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.838+0000 7f3440fa4640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f343c15f9b0 0x7f343c072130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40920/0 (socket says 172.21.15.60:40920) 2024-05-09T05:02:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.838+0000 7f3440fa4640 1 -- 172.21.15.60:0/3952924968 learned_addr learned my addr 172.21.15.60:0/3952924968 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:02:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f3441fa6640 1 --2- 172.21.15.60:0/3952924968 >> v2:172.21.15.114:3300/0 conn(0x7f343c072840 0x7f343c1450f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f343bfff640 1 -- 172.21.15.60:0/3952924968 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4243426766 0 0) 0x7f343c0cf200 con 0x7f343c15f9b0 2024-05-09T05:02:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f343bfff640 1 -- 172.21.15.60:0/3952924968 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3424003520 con 0x7f343c15f9b0 2024-05-09T05:02:08.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f343bfff640 1 -- 172.21.15.60:0/3952924968 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2104510971 0 0) 0x7f3424003520 con 0x7f343c15f9b0 2024-05-09T05:02:08.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f343bfff640 1 -- 172.21.15.60:0/3952924968 >> v2:172.21.15.114:3300/0 conn(0x7f343c072840 msgr2=0x7f343c1450f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:08.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f343bfff640 1 --2- 172.21.15.60:0/3952924968 >> v2:172.21.15.114:3300/0 conn(0x7f343c072840 0x7f343c1450f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f343bfff640 1 -- 172.21.15.60:0/3952924968 >> v1:172.21.15.114:6789/0 conn(0x7f343c162fd0 legacy=0x7f343c0ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f343bfff640 1 -- 172.21.15.60:0/3952924968 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343c160280 con 0x7f343c15f9b0 2024-05-09T05:02:08.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f343bfff640 1 -- 172.21.15.60:0/3952924968 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f34300016f0 con 0x7f343c15f9b0 2024-05-09T05:02:08.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f343bfff640 1 -- 172.21.15.60:0/3952924968 >> v1:172.21.15.60:6789/0 conn(0x7f343c15f9b0 legacy=0x7f343c072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f343bfff640 1 --2- 172.21.15.60:0/3952924968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3424003a50 0x7f3424003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f343bfff640 1 --2- 172.21.15.60:0/3952924968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3424004800 0x7f3424004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f343bfff640 1 --2- 172.21.15.60:0/3952924968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34240051c0 0x7f342400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f343bfff640 1 -- 172.21.15.60:0/3952924968 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f343c144c60 con 0x7f3424004800 2024-05-09T05:02:08.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f343bfff640 1 -- 172.21.15.60:0/3952924968 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f342400c160 con 0x7f34240051c0 2024-05-09T05:02:08.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f343bfff640 1 -- 172.21.15.60:0/3952924968 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f342400c360 con 0x7f3424003a50 2024-05-09T05:02:08.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f3440fa4640 1 --2- 172.21.15.60:0/3952924968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3424004800 0x7f3424004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f3441fa6640 1 --2- 172.21.15.60:0/3952924968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34240051c0 0x7f342400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f34417a5640 1 --2- 172.21.15.60:0/3952924968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3424003a50 0x7f3424003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f3440fa4640 1 -- 172.21.15.60:0/3952924968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3424003a50 msgr2=0x7f3424003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:08.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f3440fa4640 1 --2- 172.21.15.60:0/3952924968 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3424003a50 0x7f3424003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f3440fa4640 1 -- 172.21.15.60:0/3952924968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34240051c0 msgr2=0x7f342400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:08.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f3440fa4640 1 --2- 172.21.15.60:0/3952924968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34240051c0 0x7f342400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f3440fa4640 1 -- 172.21.15.60:0/3952924968 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3424003520 con 0x7f3424004800 2024-05-09T05:02:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.839+0000 7f3441fa6640 1 -- 172.21.15.60:0/3952924968 reap_dead start 2024-05-09T05:02:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.840+0000 7f3440fa4640 1 --2- 172.21.15.60:0/3952924968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3424004800 0x7f3424004c80 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3430001050 tx=0x7f3424005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e9710a60531194 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.840+0000 7f343bfff640 1 -- 172.21.15.60:0/3952924968 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3430007d10 con 0x7f3424004800 2024-05-09T05:02:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.840+0000 7f343bfff640 1 -- 172.21.15.60:0/3952924968 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3430005ad0 con 0x7f3424004800 2024-05-09T05:02:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.840+0000 7f343bfff640 1 -- 172.21.15.60:0/3952924968 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3430007070 con 0x7f3424004800 2024-05-09T05:02:08.846 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.841+0000 7f3443a09640 1 -- 172.21.15.60:0/3952924968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3424004800 msgr2=0x7f3424004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.841+0000 7f3443a09640 1 --2- 172.21.15.60:0/3952924968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3424004800 0x7f3424004c80 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3430001050 tx=0x7f3424005830 comp rx=0 tx=0).stop 2024-05-09T05:02:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.841+0000 7f3443a09640 1 -- 172.21.15.60:0/3952924968 shutdown_connections 2024-05-09T05:02:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.841+0000 7f3443a09640 1 --2- 172.21.15.60:0/3952924968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3424004800 0x7f3424004c80 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.841+0000 7f3443a09640 1 -- 172.21.15.60:0/3952924968 >> 172.21.15.60:0/3952924968 conn(0x7f343c0d2320 msgr2=0x7f343c0d2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.841+0000 7f3443a09640 1 -- 172.21.15.60:0/3952924968 shutdown_connections 2024-05-09T05:02:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.841+0000 7f3443a09640 1 -- 172.21.15.60:0/3952924968 wait complete. 2024-05-09T05:02:08.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.842+0000 7f3443a09640 1 Processor -- start 2024-05-09T05:02:08.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.842+0000 7f3443a09640 1 -- start start 2024-05-09T05:02:08.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.843+0000 7f3443a09640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3424004800 0x7f343c13e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.843+0000 7f3443a09640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f343c143a20 0x7f343c0cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.843+0000 7f3443a09640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f343c156d70 0x7f343c155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.843+0000 7f3443a09640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f343c072130 con 0x7f343c143a20 2024-05-09T05:02:08.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.843+0000 7f3443a09640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f343c160840 con 0x7f343c156d70 2024-05-09T05:02:08.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.843+0000 7f3443a09640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f343c0722b0 con 0x7f3424004800 2024-05-09T05:02:08.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.843+0000 7f3440fa4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f343c143a20 0x7f343c0cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.843+0000 7f3440fa4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f343c143a20 0x7f343c0cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47558/0 (socket says 172.21.15.60:47558) 2024-05-09T05:02:08.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.843+0000 7f3440fa4640 1 -- 172.21.15.60:0/3799315521 learned_addr learned my addr 172.21.15.60:0/3799315521 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:08.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.843+0000 7f3441fa6640 1 --2- 172.21.15.60:0/3799315521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f343c156d70 0x7f343c155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.843+0000 7f34417a5640 1 --2- 172.21.15.60:0/3799315521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3424004800 0x7f343c13e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.843+0000 7f3440fa4640 1 -- 172.21.15.60:0/3799315521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3424004800 msgr2=0x7f343c13e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.843+0000 7f3440fa4640 1 --2- 172.21.15.60:0/3799315521 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f3424004800 0x7f343c13e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.843+0000 7f3440fa4640 1 -- 172.21.15.60:0/3799315521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f343c156d70 msgr2=0x7f343c155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.843+0000 7f3440fa4640 1 --2- 172.21.15.60:0/3799315521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f343c156d70 0x7f343c155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.843+0000 7f3440fa4640 1 -- 172.21.15.60:0/3799315521 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343c13e870 con 0x7f343c143a20 2024-05-09T05:02:08.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.843+0000 7f3440fa4640 1 --2- 172.21.15.60:0/3799315521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f343c143a20 0x7f343c0cb570 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f343002acf0 tx=0x7f343c160450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:08.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.843+0000 7f3439ffb640 1 -- 172.21.15.60:0/3799315521 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3430041020 con 0x7f343c143a20 2024-05-09T05:02:08.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.844+0000 7f3439ffb640 1 -- 172.21.15.60:0/3799315521 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3430038040 con 0x7f343c143a20 2024-05-09T05:02:08.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.844+0000 7f3443a09640 1 -- 172.21.15.60:0/3799315521 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f343c13eb50 con 0x7f343c143a20 2024-05-09T05:02:08.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.844+0000 7f3439ffb640 1 -- 172.21.15.60:0/3799315521 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3430007070 con 0x7f343c143a20 2024-05-09T05:02:08.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.844+0000 7f3443a09640 1 -- 172.21.15.60:0/3799315521 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f343c13f110 con 0x7f343c143a20 2024-05-09T05:02:08.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.844+0000 7f3443a09640 1 -- 172.21.15.60:0/3799315521 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f343c0b6230 con 0x7f343c143a20 2024-05-09T05:02:08.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.845+0000 7f3439ffb640 1 -- 172.21.15.60:0/3799315521 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3430044020 con 0x7f343c143a20 2024-05-09T05:02:08.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.846+0000 7f3439ffb640 1 --2- 172.21.15.60:0/3799315521 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f3428076300 0x7f34280787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.846+0000 7f3439ffb640 1 -- 172.21.15.60:0/3799315521 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7f34300b7650 con 0x7f343c143a20 2024-05-09T05:02:08.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.846+0000 7f3439ffb640 1 --2- 172.21.15.60:0/3799315521 >> [v2:172.21.15.114:6800/4142062928,v1:172.21.15.114:6802/4142062928] conn(0x7f342807c7a0 0x7f342807ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.854 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.846+0000 7f3439ffb640 1 -- 172.21.15.60:0/3799315521 --> [v2:172.21.15.114:6800/4142062928,v1:172.21.15.114:6802/4142062928] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3430040e70 con 0x7f342807c7a0 2024-05-09T05:02:08.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.846+0000 7f3439ffb640 1 -- 172.21.15.60:0/3799315521 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f34300be050 con 0x7f343c143a20 2024-05-09T05:02:08.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.846+0000 7f34417a5640 1 --2- 172.21.15.60:0/3799315521 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f3428076300 0x7f34280787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.846+0000 7f3441fa6640 1 --2- 172.21.15.60:0/3799315521 >> [v2:172.21.15.114:6800/4142062928,v1:172.21.15.114:6802/4142062928] conn(0x7f342807c7a0 0x7f342807ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.847+0000 7f3441fa6640 1 --2- 172.21.15.60:0/3799315521 >> [v2:172.21.15.114:6800/4142062928,v1:172.21.15.114:6802/4142062928] conn(0x7f342807c7a0 0x7f342807ebe0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:08.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.847+0000 7f34417a5640 1 --2- 172.21.15.60:0/3799315521 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f3428076300 0x7f34280787c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f343c0ce670 tx=0x7f3434009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:08.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.847+0000 7f3439ffb640 1 -- 172.21.15.60:0/3799315521 <== osd.7 v2:172.21.15.114:6800/4142062928 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3430040e70 con 0x7f342807c7a0 2024-05-09T05:02:08.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.861+0000 7f3443a09640 1 -- 172.21.15.60:0/3799315521 --> [v2:172.21.15.114:6800/4142062928,v1:172.21.15.114:6802/4142062928] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f343c13f560 con 0x7f342807c7a0 2024-05-09T05:02:08.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.861+0000 7f3439ffb640 1 -- 172.21.15.60:0/3799315521 <== osd.7 v2:172.21.15.114:6800/4142062928 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f343c13f560 con 0x7f342807c7a0 2024-05-09T05:02:08.862 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-05-09T05:02:08.862 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T05:02:08.862 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-05-09T05:02:08.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.861+0000 7f3443a09640 1 -- 172.21.15.60:0/3799315521 >> [v2:172.21.15.114:6800/4142062928,v1:172.21.15.114:6802/4142062928] conn(0x7f342807c7a0 msgr2=0x7f342807ebe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.862+0000 7f3443a09640 1 --2- 172.21.15.60:0/3799315521 >> [v2:172.21.15.114:6800/4142062928,v1:172.21.15.114:6802/4142062928] conn(0x7f342807c7a0 0x7f342807ebe0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.862+0000 7f3443a09640 1 -- 172.21.15.60:0/3799315521 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f3428076300 msgr2=0x7f34280787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.862+0000 7f3443a09640 1 --2- 172.21.15.60:0/3799315521 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f3428076300 0x7f34280787c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f343c0ce670 tx=0x7f3434009040 comp rx=0 tx=0).stop 2024-05-09T05:02:08.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.862+0000 7f3443a09640 1 -- 172.21.15.60:0/3799315521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f343c143a20 msgr2=0x7f343c0cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.862+0000 7f3443a09640 1 --2- 172.21.15.60:0/3799315521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f343c143a20 0x7f343c0cb570 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f343002acf0 tx=0x7f343c160450 comp rx=0 tx=0).stop 2024-05-09T05:02:08.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.862+0000 7f3441fa6640 1 -- 172.21.15.60:0/3799315521 reap_dead start 2024-05-09T05:02:08.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.862+0000 7f3443a09640 1 -- 172.21.15.60:0/3799315521 shutdown_connections 2024-05-09T05:02:08.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.862+0000 7f3443a09640 1 -- 172.21.15.60:0/3799315521 >> 172.21.15.60:0/3799315521 conn(0x7f343c0d2320 msgr2=0x7f343c0b5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:08.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.862+0000 7f3443a09640 1 -- 172.21.15.60:0/3799315521 shutdown_connections 2024-05-09T05:02:08.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.862+0000 7f3443a09640 1 -- 172.21.15.60:0/3799315521 wait complete. 2024-05-09T05:02:08.873 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-05-09T05:02:08.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.976+0000 7fa14f0a7640 1 Processor -- start 2024-05-09T05:02:08.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.976+0000 7fa14f0a7640 1 -- start start 2024-05-09T05:02:08.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.977+0000 7fa14f0a7640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7fa14815f300 0x7fa1481639f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.977+0000 7fa14f0a7640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa14806f9f0 0x7fa1480d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.977+0000 7fa14f0a7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1480d3930 0x7fa14806ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.977+0000 7fa14f0a7640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7fa14806ff60 con 0x7fa14806f9f0 2024-05-09T05:02:08.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.977+0000 7fa14f0a7640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7fa148070240 con 0x7fa14815f300 2024-05-09T05:02:08.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.977+0000 7fa14f0a7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa148070520 con 0x7fa1480d3930 2024-05-09T05:02:08.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.977+0000 7fa147fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa14806f9f0 0x7fa1480d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.977+0000 7fa147fff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa14806f9f0 0x7fa1480d33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47564/0 (socket says 172.21.15.60:47564) 2024-05-09T05:02:08.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.977+0000 7fa147fff640 1 -- 172.21.15.60:0/69553273 learned_addr learned my addr 172.21.15.60:0/69553273 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:08.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.977+0000 7fa14ce43640 1 --2- 172.21.15.60:0/69553273 >> v2:172.21.15.114:3300/0 conn(0x7fa14815f300 0x7fa1481639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.977+0000 7fa14d644640 1 --2- 172.21.15.60:0/69553273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1480d3930 0x7fa14806ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.978+0000 7fa147fff640 1 -- 172.21.15.60:0/69553273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1480d3930 msgr2=0x7fa14806ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:08.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.978+0000 7fa147fff640 1 --2- 172.21.15.60:0/69553273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1480d3930 0x7fa14806ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.978+0000 7fa147fff640 1 -- 172.21.15.60:0/69553273 >> v2:172.21.15.114:3300/0 conn(0x7fa14815f300 msgr2=0x7fa1481639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:08.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.978+0000 7fa147fff640 1 --2- 172.21.15.60:0/69553273 >> v2:172.21.15.114:3300/0 conn(0x7fa14815f300 0x7fa1481639f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.978+0000 7fa147fff640 1 -- 172.21.15.60:0/69553273 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa14806e740 con 0x7fa14806f9f0 2024-05-09T05:02:08.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.978+0000 7fa14ce43640 1 --2- 172.21.15.60:0/69553273 >> v2:172.21.15.114:3300/0 conn(0x7fa14815f300 0x7fa1481639f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:02:08.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.978+0000 7fa147fff640 1 --2- 172.21.15.60:0/69553273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa14806f9f0 0x7fa1480d33f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa148070730 tx=0x7fa13002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5200ef12ae9ced8f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:08.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.978+0000 7fa1477fe640 1 -- 172.21.15.60:0/69553273 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa130036070 con 0x7fa14806f9f0 2024-05-09T05:02:08.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.979+0000 7fa1477fe640 1 -- 172.21.15.60:0/69553273 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa130031600 con 0x7fa14806f9f0 2024-05-09T05:02:08.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.979+0000 7fa1477fe640 1 -- 172.21.15.60:0/69553273 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1300319d0 con 0x7fa14806f9f0 2024-05-09T05:02:08.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.979+0000 7fa14f0a7640 1 -- 172.21.15.60:0/69553273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa14806f9f0 msgr2=0x7fa1480d33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.983 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.979+0000 7fa14f0a7640 1 --2- 172.21.15.60:0/69553273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa14806f9f0 0x7fa1480d33f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa148070730 tx=0x7fa13002a680 comp rx=0 tx=0).stop 2024-05-09T05:02:08.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.979+0000 7fa14f0a7640 1 -- 172.21.15.60:0/69553273 shutdown_connections 2024-05-09T05:02:08.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.979+0000 7fa14f0a7640 1 --2- 172.21.15.60:0/69553273 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa1480d3930 0x7fa14806ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.979+0000 7fa14f0a7640 1 --2- 172.21.15.60:0/69553273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa14806f9f0 0x7fa1480d33f0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.979+0000 7fa14f0a7640 1 --2- 172.21.15.60:0/69553273 >> v2:172.21.15.114:3300/0 conn(0x7fa14815f300 0x7fa1481639f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.979+0000 7fa14f0a7640 1 -- 172.21.15.60:0/69553273 >> 172.21.15.60:0/69553273 conn(0x7fa14809b850 msgr2=0x7fa14809bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:08.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.979+0000 7fa14f0a7640 1 -- 172.21.15.60:0/69553273 shutdown_connections 2024-05-09T05:02:08.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.979+0000 7fa14f0a7640 1 -- 172.21.15.60:0/69553273 wait complete. 2024-05-09T05:02:08.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.980+0000 7fa14f0a7640 1 Processor -- start 2024-05-09T05:02:08.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.980+0000 7fa14f0a7640 1 -- start start 2024-05-09T05:02:08.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.981+0000 7fa14f0a7640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa14806f9f0 0x7fa148161850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.981+0000 7fa14f0a7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1480d3930 0x7fa1480aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.981+0000 7fa14f0a7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa14815f300 0x7fa14815ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.981+0000 7fa14f0a7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7fa14806f480 con 0x7fa14815f300 2024-05-09T05:02:08.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.981+0000 7fa14f0a7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa148163f30 con 0x7fa1480d3930 2024-05-09T05:02:08.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.981+0000 7fa14f0a7640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7fa14806f600 con 0x7fa14806f9f0 2024-05-09T05:02:08.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.981+0000 7fa14d644640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa14815f300 0x7fa14815ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.981+0000 7fa14d644640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa14815f300 0x7fa14815ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47578/0 (socket says 172.21.15.60:47578) 2024-05-09T05:02:08.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.981+0000 7fa14d644640 1 -- 172.21.15.60:0/1500727904 learned_addr learned my addr 172.21.15.60:0/1500727904 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:08.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.981+0000 7fa14ce43640 1 --2- 172.21.15.60:0/1500727904 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa14806f9f0 0x7fa148161850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.981+0000 7fa147fff640 1 --2- 172.21.15.60:0/1500727904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1480d3930 0x7fa1480aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.982+0000 7fa14d644640 1 -- 172.21.15.60:0/1500727904 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa14806f9f0 msgr2=0x7fa148161850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.982+0000 7fa14d644640 1 --2- 172.21.15.60:0/1500727904 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa14806f9f0 0x7fa148161850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.982+0000 7fa14d644640 1 -- 172.21.15.60:0/1500727904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1480d3930 msgr2=0x7fa1480aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:08.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.982+0000 7fa14d644640 1 --2- 172.21.15.60:0/1500727904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1480d3930 0x7fa1480aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:08.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.982+0000 7fa14d644640 1 -- 172.21.15.60:0/1500727904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa148160820 con 0x7fa14815f300 2024-05-09T05:02:08.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.982+0000 7fa147fff640 1 --2- 172.21.15.60:0/1500727904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1480d3930 0x7fa1480aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:02:08.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.982+0000 7fa14ce43640 1 --2- 172.21.15.60:0/1500727904 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa14806f9f0 0x7fa148161850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:02:08.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.982+0000 7fa14d644640 1 --2- 172.21.15.60:0/1500727904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa14815f300 0x7fa14815ae70 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa14000d800 tx=0x7fa14000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:08.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.982+0000 7fa1457fa640 1 -- 172.21.15.60:0/1500727904 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa140018020 con 0x7fa14815f300 2024-05-09T05:02:08.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.982+0000 7fa1457fa640 1 -- 172.21.15.60:0/1500727904 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1400096c0 con 0x7fa14815f300 2024-05-09T05:02:08.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.982+0000 7fa14f0a7640 1 -- 172.21.15.60:0/1500727904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa148160b00 con 0x7fa14815f300 2024-05-09T05:02:08.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.982+0000 7fa1457fa640 1 -- 172.21.15.60:0/1500727904 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa140009a90 con 0x7fa14815f300 2024-05-09T05:02:08.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.982+0000 7fa14f0a7640 1 -- 172.21.15.60:0/1500727904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa148161d90 con 0x7fa14815f300 2024-05-09T05:02:08.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.983+0000 7fa14f0a7640 1 -- 172.21.15.60:0/1500727904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa148162450 con 0x7fa14815f300 2024-05-09T05:02:08.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.984+0000 7fa1457fa640 1 -- 172.21.15.60:0/1500727904 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa14001b020 con 0x7fa14815f300 2024-05-09T05:02:08.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.984+0000 7fa1457fa640 1 --2- 172.21.15.60:0/1500727904 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa128076300 0x7fa1280787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:08.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.984+0000 7fa1457fa640 1 -- 172.21.15.60:0/1500727904 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7fa140097670 con 0x7fa14815f300 2024-05-09T05:02:08.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.984+0000 7fa14ce43640 1 --2- 172.21.15.60:0/1500727904 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa128076300 0x7fa1280787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:08.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.985+0000 7fa14ce43640 1 --2- 172.21.15.60:0/1500727904 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa128076300 0x7fa1280787c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa148099370 tx=0x7fa138008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:08.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:08.988+0000 7fa1457fa640 1 -- 172.21.15.60:0/1500727904 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa148162450 con 0x7fa14815f300 2024-05-09T05:02:09.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.179+0000 7fa14f0a7640 1 -- 172.21.15.60:0/1500727904 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fa148161020 con 0x7fa128076300 2024-05-09T05:02:09.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.180+0000 7fa1457fa640 1 -- 172.21.15.60:0/1500727904 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa13800f830 con 0x7fa128076300 2024-05-09T05:02:09.180 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 7 to deep-scrub 2024-05-09T05:02:09.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.182+0000 7fa14f0a7640 1 -- 172.21.15.60:0/1500727904 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa128076300 msgr2=0x7fa1280787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:09.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.182+0000 7fa14f0a7640 1 --2- 172.21.15.60:0/1500727904 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa128076300 0x7fa1280787c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa148099370 tx=0x7fa138008040 comp rx=0 tx=0).stop 2024-05-09T05:02:09.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.182+0000 7fa14f0a7640 1 -- 172.21.15.60:0/1500727904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa14815f300 msgr2=0x7fa14815ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:09.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.182+0000 7fa14f0a7640 1 --2- 172.21.15.60:0/1500727904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa14815f300 0x7fa14815ae70 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa14000d800 tx=0x7fa14000dcc0 comp rx=0 tx=0).stop 2024-05-09T05:02:09.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.182+0000 7fa14f0a7640 1 -- 172.21.15.60:0/1500727904 shutdown_connections 2024-05-09T05:02:09.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.182+0000 7fa14f0a7640 1 --2- 172.21.15.60:0/1500727904 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7fa128076300 0x7fa1280787c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:09.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.182+0000 7fa14f0a7640 1 --2- 172.21.15.60:0/1500727904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa14815f300 0x7fa14815ae70 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:09.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.182+0000 7fa14f0a7640 1 --2- 172.21.15.60:0/1500727904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1480d3930 0x7fa1480aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:09.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.182+0000 7fa14f0a7640 1 --2- 172.21.15.60:0/1500727904 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7fa14806f9f0 0x7fa148161850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:09.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.182+0000 7fa14f0a7640 1 -- 172.21.15.60:0/1500727904 >> 172.21.15.60:0/1500727904 conn(0x7fa14809b850 msgr2=0x7fa148162950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:09.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.182+0000 7fa14f0a7640 1 -- 172.21.15.60:0/1500727904 shutdown_connections 2024-05-09T05:02:09.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.182+0000 7fa14f0a7640 1 -- 172.21.15.60:0/1500727904 wait complete. 2024-05-09T05:02:09.194 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:02:09.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.298+0000 7f0d9e609640 1 Processor -- start 2024-05-09T05:02:09.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.298+0000 7f0d9e609640 1 -- start start 2024-05-09T05:02:09.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.298+0000 7f0d9e609640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0d98073fb0 0x7f0d980b4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:09.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.298+0000 7f0d9e609640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f0d980b4b60 0x7f0d98072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:09.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.298+0000 7f0d9e609640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d9815f7a0 con 0x7f0d9815f3c0 2024-05-09T05:02:09.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.298+0000 7f0d9e609640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f0d98071cb0 con 0x7f0d98073fb0 2024-05-09T05:02:09.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.298+0000 7f0d9e609640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f0d980b4f90 con 0x7f0d980b4b60 2024-05-09T05:02:09.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.299+0000 7f0d977fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0d98073fb0 0x7f0d980b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:09.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.299+0000 7f0d97fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0d9815f3c0 0x7f0d9809ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40936/0 (socket says 172.21.15.60:40936) 2024-05-09T05:02:09.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.299+0000 7f0d977fe640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0d98073fb0 0x7f0d980b4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47584/0 (socket says 172.21.15.60:47584) 2024-05-09T05:02:09.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.299+0000 7f0d97fff640 1 -- 172.21.15.60:0/4133000543 learned_addr learned my addr 172.21.15.60:0/4133000543 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:02:09.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.299+0000 7f0d9cba6640 1 --2- 172.21.15.60:0/4133000543 >> v2:172.21.15.114:3300/0 conn(0x7f0d980b4b60 0x7f0d98072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:09.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.299+0000 7f0d977fe640 1 -- 172.21.15.60:0/4133000543 >> v2:172.21.15.114:3300/0 conn(0x7f0d980b4b60 msgr2=0x7f0d98072140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:09.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.299+0000 7f0d977fe640 1 --2- 172.21.15.60:0/4133000543 >> v2:172.21.15.114:3300/0 conn(0x7f0d980b4b60 0x7f0d98072140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:09.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.299+0000 7f0d977fe640 1 -- 172.21.15.60:0/4133000543 >> v1:172.21.15.60:6789/0 conn(0x7f0d9815f3c0 legacy=0x7f0d9809ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:09.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.299+0000 7f0d977fe640 1 -- 172.21.15.60:0/4133000543 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d9809f480 con 0x7f0d98073fb0 2024-05-09T05:02:09.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.299+0000 7f0d96ffd640 1 -- 172.21.15.60:0/4133000543 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2963372374 0 0) 0x7f0d9815f7a0 con 0x7f0d9815f3c0 2024-05-09T05:02:09.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.300+0000 7f0d977fe640 1 --2- 172.21.15.60:0/4133000543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d98073fb0 0x7f0d980b4620 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f0d980d52d0 tx=0x7f0d8c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b6ec15c9b24c0c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:09.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.300+0000 7f0d96ffd640 1 -- 172.21.15.60:0/4133000543 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c037070 con 0x7f0d98073fb0 2024-05-09T05:02:09.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.300+0000 7f0d96ffd640 1 -- 172.21.15.60:0/4133000543 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d8c032440 con 0x7f0d98073fb0 2024-05-09T05:02:09.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.300+0000 7f0d96ffd640 1 -- 172.21.15.60:0/4133000543 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c02ae60 con 0x7f0d98073fb0 2024-05-09T05:02:09.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.300+0000 7f0d9e609640 1 -- 172.21.15.60:0/4133000543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d98073fb0 msgr2=0x7f0d980b4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:09.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.300+0000 7f0d9e609640 1 --2- 172.21.15.60:0/4133000543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d98073fb0 0x7f0d980b4620 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f0d980d52d0 tx=0x7f0d8c02a110 comp rx=0 tx=0).stop 2024-05-09T05:02:09.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.300+0000 7f0d9e609640 1 -- 172.21.15.60:0/4133000543 shutdown_connections 2024-05-09T05:02:09.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.300+0000 7f0d9e609640 1 --2- 172.21.15.60:0/4133000543 >> v2:172.21.15.114:3300/0 conn(0x7f0d980b4b60 0x7f0d98072140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:09.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.300+0000 7f0d9e609640 1 --2- 172.21.15.60:0/4133000543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d98073fb0 0x7f0d980b4620 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:09.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.300+0000 7f0d9e609640 1 -- 172.21.15.60:0/4133000543 >> 172.21.15.60:0/4133000543 conn(0x7f0d980d1470 msgr2=0x7f0d980d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:09.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.300+0000 7f0d9e609640 1 -- 172.21.15.60:0/4133000543 shutdown_connections 2024-05-09T05:02:09.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.300+0000 7f0d9e609640 1 -- 172.21.15.60:0/4133000543 wait complete. 2024-05-09T05:02:09.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.302+0000 7f0d9e609640 1 Processor -- start 2024-05-09T05:02:09.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.302+0000 7f0d9e609640 1 -- start start 2024-05-09T05:02:09.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d9e609640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98073fb0 0x7f0d980acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:09.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d9e609640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d980b4b60 0x7f0d980d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:09.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d9e609640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d9815f3c0 0x7f0d9815d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:09.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d9e609640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f0d98072680 con 0x7f0d980b4b60 2024-05-09T05:02:09.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d9e609640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0d98071940 con 0x7f0d98073fb0 2024-05-09T05:02:09.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d9e609640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f0d98072800 con 0x7f0d9815f3c0 2024-05-09T05:02:09.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d977fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d980b4b60 0x7f0d980d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:09.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d977fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d980b4b60 0x7f0d980d41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47594/0 (socket says 172.21.15.60:47594) 2024-05-09T05:02:09.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d977fe640 1 -- 172.21.15.60:0/3942945604 learned_addr learned my addr 172.21.15.60:0/3942945604 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:09.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d97fff640 1 --2- 172.21.15.60:0/3942945604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98073fb0 0x7f0d980acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:09.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d977fe640 1 -- 172.21.15.60:0/3942945604 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d9815f3c0 msgr2=0x7f0d9815d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:09.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d977fe640 1 --2- 172.21.15.60:0/3942945604 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d9815f3c0 0x7f0d9815d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:09.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d977fe640 1 -- 172.21.15.60:0/3942945604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98073fb0 msgr2=0x7f0d980acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:09.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d977fe640 1 --2- 172.21.15.60:0/3942945604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98073fb0 0x7f0d980acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:09.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d977fe640 1 -- 172.21.15.60:0/3942945604 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d980aefb0 con 0x7f0d980b4b60 2024-05-09T05:02:09.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d977fe640 1 --2- 172.21.15.60:0/3942945604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d980b4b60 0x7f0d980d41e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c000c00 tx=0x7f0d8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:09.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d94ff9640 1 -- 172.21.15.60:0/3942945604 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c037040 con 0x7f0d980b4b60 2024-05-09T05:02:09.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d94ff9640 1 -- 172.21.15.60:0/3942945604 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d8c042070 con 0x7f0d980b4b60 2024-05-09T05:02:09.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d94ff9640 1 -- 172.21.15.60:0/3942945604 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c030bd0 con 0x7f0d980b4b60 2024-05-09T05:02:09.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d9e609640 1 -- 172.21.15.60:0/3942945604 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d980af290 con 0x7f0d980b4b60 2024-05-09T05:02:09.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.303+0000 7f0d9e609640 1 -- 172.21.15.60:0/3942945604 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d980af850 con 0x7f0d980b4b60 2024-05-09T05:02:09.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.304+0000 7f0d9e609640 1 -- 172.21.15.60:0/3942945604 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d98051040 con 0x7f0d980b4b60 2024-05-09T05:02:09.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.305+0000 7f0d94ff9640 1 -- 172.21.15.60:0/3942945604 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0d8c047460 con 0x7f0d980b4b60 2024-05-09T05:02:09.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.306+0000 7f0d94ff9640 1 --2- 172.21.15.60:0/3942945604 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0d840762d0 0x7f0d84078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:09.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.306+0000 7f0d94ff9640 1 -- 172.21.15.60:0/3942945604 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7f0d8c0b80f0 con 0x7f0d980b4b60 2024-05-09T05:02:09.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.306+0000 7f0d97fff640 1 --2- 172.21.15.60:0/3942945604 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0d840762d0 0x7f0d84078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:09.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.307+0000 7f0d97fff640 1 --2- 172.21.15.60:0/3942945604 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0d840762d0 0x7f0d84078790 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0d88000f60 tx=0x7f0d88009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:09.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.310+0000 7f0d94ff9640 1 -- 172.21.15.60:0/3942945604 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0d8c032e20 con 0x7f0d980b4b60 2024-05-09T05:02:09.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.501+0000 7f0d9e609640 1 -- 172.21.15.60:0/3942945604 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0d980996d0 con 0x7f0d840762d0 2024-05-09T05:02:09.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.502+0000 7f0d94ff9640 1 -- 172.21.15.60:0/3942945604 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13399 (secure 0 0 0) 0x7f0d980996d0 con 0x7f0d840762d0 2024-05-09T05:02:09.503 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:02:09.503 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-05-09T05:02:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.504+0000 7f0d9e609640 1 -- 172.21.15.60:0/3942945604 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0d840762d0 msgr2=0x7f0d84078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.504+0000 7f0d9e609640 1 --2- 172.21.15.60:0/3942945604 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0d840762d0 0x7f0d84078790 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0d88000f60 tx=0x7f0d88009040 comp rx=0 tx=0).stop 2024-05-09T05:02:09.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.504+0000 7f0d9e609640 1 -- 172.21.15.60:0/3942945604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d980b4b60 msgr2=0x7f0d980d41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.504+0000 7f0d9e609640 1 --2- 172.21.15.60:0/3942945604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d980b4b60 0x7f0d980d41e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c000c00 tx=0x7f0d8c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:02:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.504+0000 7f0d9e609640 1 -- 172.21.15.60:0/3942945604 shutdown_connections 2024-05-09T05:02:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.504+0000 7f0d9e609640 1 --2- 172.21.15.60:0/3942945604 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f0d840762d0 0x7f0d84078790 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.504+0000 7f0d9e609640 1 --2- 172.21.15.60:0/3942945604 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f0d9815f3c0 0x7f0d9815d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.504+0000 7f0d9e609640 1 --2- 172.21.15.60:0/3942945604 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d980b4b60 0x7f0d980d41e0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.504+0000 7f0d9e609640 1 --2- 172.21.15.60:0/3942945604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98073fb0 0x7f0d980acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.505+0000 7f0d9e609640 1 -- 172.21.15.60:0/3942945604 >> 172.21.15.60:0/3942945604 conn(0x7f0d980d1470 msgr2=0x7f0d9815e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:09.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.505+0000 7f0d9e609640 1 -- 172.21.15.60:0/3942945604 shutdown_connections 2024-05-09T05:02:09.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:09.505+0000 7f0d9e609640 1 -- 172.21.15.60:0/3942945604 wait complete. 2024-05-09T05:02:09.516 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-05-09T05:02:08.935883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":239928,"kb_used_data":7112,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734792,"statfs":{"total":773094113280,"available":772848427008,"internally_reserved":0,"allocated":7282688,"data_stored":3969120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.965188"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":188,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-09T05:02:01.660678+0000","last_change":"2024-05-09T05:00:08.667158+0000","last_active":"2024-05-09T05:02:01.660678+0000","last_peered":"2024-05-09T05:02:01.660678+0000","last_clean":"2024-05-09T05:02:01.660678+0000","last_became_active":"2024-05-09T05:00:08.666316+0000","last_became_peered":"2024-05-09T05:00:08.666316+0000","last_unstale":"2024-05-09T05:02:01.660678+0000","last_undegraded":"2024-05-09T05:02:01.660678+0000","last_fullsized":"2024-05-09T05:02:01.660678+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:00:07.370119+0000","last_clean_scrub_stamp":"2024-05-09T05:00:07.370119+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:25:20.234643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28344,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343496,"statfs":{"total":96636764160,"available":96607739904,"internally_reserved":0,"allocated":1355776,"data_stored":938916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":761856,"data_stored":348548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27768,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":761856,"data_stored":348548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27768,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":761856,"data_stored":348548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":761856,"data_stored":348548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31924,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339916,"statfs":{"total":96636764160,"available":96604073984,"internally_reserved":0,"allocated":761856,"data_stored":348548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31924,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339916,"statfs":{"total":96636764160,"available":96604073984,"internally_reserved":0,"allocated":761856,"data_stored":348548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1355776,"data_stored":938916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:02:09.518 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-05-09T05:00:07.370119+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=0, tm_sec=7, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=2, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:02:09.519 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-05-09T05:02:22.918 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:02:22.922 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:02:29.520 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:02:29.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.637+0000 7ff637459640 1 Processor -- start 2024-05-09T05:02:29.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.638+0000 7ff637459640 1 -- start start 2024-05-09T05:02:29.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.638+0000 7ff637459640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff63015d3b0 0x7ff63015d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:29.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.638+0000 7ff637459640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff630074130 0x7ff63006faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:29.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.639+0000 7ff637459640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff63006ea00 con 0x7ff630074750 2024-05-09T05:02:29.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.639+0000 7ff637459640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7ff63006e7b0 con 0x7ff63015d3b0 2024-05-09T05:02:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.639+0000 7ff637459640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff63006f610 con 0x7ff630074130 2024-05-09T05:02:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.639+0000 7ff6351f5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff63015d3b0 0x7ff63015d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.639+0000 7ff6351f5640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff63015d3b0 0x7ff63015d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59200/0 (socket says 172.21.15.60:59200) 2024-05-09T05:02:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.639+0000 7ff6351f5640 1 -- 172.21.15.60:0/149602324 learned_addr learned my addr 172.21.15.60:0/149602324 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.639+0000 7ff6359f6640 1 --2- 172.21.15.60:0/149602324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff630074130 0x7ff63006faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.639+0000 7ff6351f5640 1 -- 172.21.15.60:0/149602324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff630074130 msgr2=0x7ff63006faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.639+0000 7ff6351f5640 1 --2- 172.21.15.60:0/149602324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff630074130 0x7ff63006faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.639+0000 7ff6351f5640 1 -- 172.21.15.60:0/149602324 >> v1:172.21.15.114:6789/0 conn(0x7ff630074750 legacy=0x7ff630073a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:29.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.639+0000 7ff6351f5640 1 -- 172.21.15.60:0/149602324 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff63006ece0 con 0x7ff63015d3b0 2024-05-09T05:02:29.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.640+0000 7ff6351f5640 1 --2- 172.21.15.60:0/149602324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff63015d3b0 0x7ff63015d790 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff62c009870 tx=0x7ff62c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59189566d59046c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:29.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.640+0000 7ff627fff640 1 -- 172.21.15.60:0/149602324 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c037070 con 0x7ff63015d3b0 2024-05-09T05:02:29.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.640+0000 7ff627fff640 1 -- 172.21.15.60:0/149602324 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff62c032440 con 0x7ff63015d3b0 2024-05-09T05:02:29.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.640+0000 7ff627fff640 1 -- 172.21.15.60:0/149602324 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c032810 con 0x7ff63015d3b0 2024-05-09T05:02:29.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.641+0000 7ff637459640 1 -- 172.21.15.60:0/149602324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff63015d3b0 msgr2=0x7ff63015d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:29.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.641+0000 7ff637459640 1 --2- 172.21.15.60:0/149602324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff63015d3b0 0x7ff63015d790 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff62c009870 tx=0x7ff62c02a1c0 comp rx=0 tx=0).stop 2024-05-09T05:02:29.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.641+0000 7ff637459640 1 -- 172.21.15.60:0/149602324 shutdown_connections 2024-05-09T05:02:29.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.641+0000 7ff637459640 1 --2- 172.21.15.60:0/149602324 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff630074130 0x7ff63006faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:29.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.641+0000 7ff637459640 1 --2- 172.21.15.60:0/149602324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff63015d3b0 0x7ff63015d790 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:29.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.641+0000 7ff637459640 1 -- 172.21.15.60:0/149602324 >> 172.21.15.60:0/149602324 conn(0x7ff6300972f0 msgr2=0x7ff6300976d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:29.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.641+0000 7ff637459640 1 -- 172.21.15.60:0/149602324 shutdown_connections 2024-05-09T05:02:29.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.641+0000 7ff637459640 1 -- 172.21.15.60:0/149602324 wait complete. 2024-05-09T05:02:29.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.642+0000 7ff637459640 1 Processor -- start 2024-05-09T05:02:29.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.642+0000 7ff637459640 1 -- start start 2024-05-09T05:02:29.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.643+0000 7ff637459640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff630074130 0x7ff6300a8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:29.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.643+0000 7ff637459640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff630074750 0x7ff630149f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:29.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.643+0000 7ff637459640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63015d3b0 0x7ff6300aeac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:29.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.643+0000 7ff637459640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7ff6300b4540 con 0x7ff630074130 2024-05-09T05:02:29.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.643+0000 7ff637459640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff63006f2a0 con 0x7ff63015d3b0 2024-05-09T05:02:29.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.643+0000 7ff637459640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7ff63015dcd0 con 0x7ff630074750 2024-05-09T05:02:29.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.643+0000 7ff6351f5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff630074130 0x7ff6300a8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:29.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.643+0000 7ff6351f5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff630074130 0x7ff6300a8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59204/0 (socket says 172.21.15.60:59204) 2024-05-09T05:02:29.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.643+0000 7ff6351f5640 1 -- 172.21.15.60:0/1489309587 learned_addr learned my addr 172.21.15.60:0/1489309587 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:29.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.643+0000 7ff6359f6640 1 --2- 172.21.15.60:0/1489309587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63015d3b0 0x7ff6300aeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:29.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.643+0000 7ff6349f4640 1 --2- 172.21.15.60:0/1489309587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff630074750 0x7ff630149f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:29.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.643+0000 7ff6351f5640 1 -- 172.21.15.60:0/1489309587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff630074750 msgr2=0x7ff630149f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:29.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.643+0000 7ff6351f5640 1 --2- 172.21.15.60:0/1489309587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff630074750 0x7ff630149f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:29.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.643+0000 7ff6351f5640 1 -- 172.21.15.60:0/1489309587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63015d3b0 msgr2=0x7ff6300aeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:29.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.643+0000 7ff6351f5640 1 --2- 172.21.15.60:0/1489309587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63015d3b0 0x7ff6300aeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:29.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.643+0000 7ff6351f5640 1 -- 172.21.15.60:0/1489309587 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff63015a2e0 con 0x7ff630074130 2024-05-09T05:02:29.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.644+0000 7ff6351f5640 1 --2- 172.21.15.60:0/1489309587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff630074130 0x7ff6300a8450 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ff62c000c00 tx=0x7ff62c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:29.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.644+0000 7ff625ffb640 1 -- 172.21.15.60:0/1489309587 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c041020 con 0x7ff630074130 2024-05-09T05:02:29.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.644+0000 7ff625ffb640 1 -- 172.21.15.60:0/1489309587 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff62c0306c0 con 0x7ff630074130 2024-05-09T05:02:29.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.644+0000 7ff637459640 1 -- 172.21.15.60:0/1489309587 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff63014a450 con 0x7ff630074130 2024-05-09T05:02:29.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.644+0000 7ff625ffb640 1 -- 172.21.15.60:0/1489309587 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c037040 con 0x7ff630074130 2024-05-09T05:02:29.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.644+0000 7ff637459640 1 -- 172.21.15.60:0/1489309587 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6301475c0 con 0x7ff630074130 2024-05-09T05:02:29.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.645+0000 7ff637459640 1 -- 172.21.15.60:0/1489309587 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff630051040 con 0x7ff630074130 2024-05-09T05:02:29.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.646+0000 7ff625ffb640 1 -- 172.21.15.60:0/1489309587 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff62c044020 con 0x7ff630074130 2024-05-09T05:02:29.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.647+0000 7ff625ffb640 1 --2- 172.21.15.60:0/1489309587 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff6140762d0 0x7ff614078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:29.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.647+0000 7ff625ffb640 1 -- 172.21.15.60:0/1489309587 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7ff62c0b7b70 con 0x7ff630074130 2024-05-09T05:02:29.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.647+0000 7ff6349f4640 1 --2- 172.21.15.60:0/1489309587 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff6140762d0 0x7ff614078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:29.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.647+0000 7ff6349f4640 1 --2- 172.21.15.60:0/1489309587 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff6140762d0 0x7ff614078790 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff6300997c0 tx=0x7ff620009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:29.650 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.651+0000 7ff625ffb640 1 -- 172.21.15.60:0/1489309587 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff62c04b030 con 0x7ff630074130 2024-05-09T05:02:29.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.841+0000 7ff637459640 1 -- 172.21.15.60:0/1489309587 --> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff630070a70 con 0x7ff6140762d0 2024-05-09T05:02:29.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.842+0000 7ff625ffb640 1 -- 172.21.15.60:0/1489309587 <== mgr.4105 v2:172.21.15.114:6832/4075088736 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13344 (secure 0 0 0) 0x7ff630070a70 con 0x7ff6140762d0 2024-05-09T05:02:29.841 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:02:29.842 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-05-09T05:02:29.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.844+0000 7ff637459640 1 -- 172.21.15.60:0/1489309587 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff6140762d0 msgr2=0x7ff614078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:29.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.844+0000 7ff637459640 1 --2- 172.21.15.60:0/1489309587 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff6140762d0 0x7ff614078790 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff6300997c0 tx=0x7ff620009040 comp rx=0 tx=0).stop 2024-05-09T05:02:29.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.844+0000 7ff637459640 1 -- 172.21.15.60:0/1489309587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff630074130 msgr2=0x7ff6300a8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:29.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.844+0000 7ff637459640 1 --2- 172.21.15.60:0/1489309587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff630074130 0x7ff6300a8450 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ff62c000c00 tx=0x7ff62c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:02:29.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.844+0000 7ff637459640 1 -- 172.21.15.60:0/1489309587 shutdown_connections 2024-05-09T05:02:29.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.844+0000 7ff637459640 1 --2- 172.21.15.60:0/1489309587 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7ff6140762d0 0x7ff614078790 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:29.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.844+0000 7ff637459640 1 --2- 172.21.15.60:0/1489309587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff63015d3b0 0x7ff6300aeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:29.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.844+0000 7ff637459640 1 --2- 172.21.15.60:0/1489309587 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7ff630074750 0x7ff630149f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:29.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.844+0000 7ff637459640 1 --2- 172.21.15.60:0/1489309587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff630074130 0x7ff6300a8450 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:29.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.844+0000 7ff637459640 1 -- 172.21.15.60:0/1489309587 >> 172.21.15.60:0/1489309587 conn(0x7ff6300972f0 msgr2=0x7ff63015c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:29.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.844+0000 7ff637459640 1 -- 172.21.15.60:0/1489309587 shutdown_connections 2024-05-09T05:02:29.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.844+0000 7ff637459640 1 -- 172.21.15.60:0/1489309587 wait complete. 2024-05-09T05:02:29.854 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-05-09T05:02:28.939636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":239920,"kb_used_data":7112,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734800,"statfs":{"total":773094113280,"available":772848435200,"internally_reserved":0,"allocated":7282688,"data_stored":3969120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002165"},"pg_stats":[{"pgid":"1.0","version":"62'74","reported_seq":200,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-05-09T05:02:09.977454+0000","last_change":"2024-05-09T05:02:09.977454+0000","last_active":"2024-05-09T05:02:09.977454+0000","last_peered":"2024-05-09T05:02:09.977454+0000","last_clean":"2024-05-09T05:02:09.977454+0000","last_became_active":"2024-05-09T05:00:08.666316+0000","last_became_peered":"2024-05-09T05:00:08.666316+0000","last_unstale":"2024-05-09T05:02:09.977454+0000","last_undegraded":"2024-05-09T05:02:09.977454+0000","last_fullsized":"2024-05-09T05:02:09.977454+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'74","last_scrub_stamp":"2024-05-09T05:02:09.977327+0000","last_deep_scrub":"62'74","last_deep_scrub_stamp":"2024-05-09T05:02:09.977327+0000","last_clean_scrub_stamp":"2024-05-09T05:02:09.977327+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:59:13.565099+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28340,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343500,"statfs":{"total":96636764160,"available":96607744000,"internally_reserved":0,"allocated":1355776,"data_stored":938916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":761856,"data_stored":348548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27768,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":761856,"data_stored":348548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27768,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":761856,"data_stored":348548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31932,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":761856,"data_stored":348548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31924,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339916,"statfs":{"total":96636764160,"available":96604073984,"internally_reserved":0,"allocated":761856,"data_stored":348548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31924,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339916,"statfs":{"total":96636764160,"available":96604073984,"internally_reserved":0,"allocated":761856,"data_stored":348548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32500,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1355776,"data_stored":938916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:02:29.855 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-05-09T05:02:29.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.959+0000 7f48881e4640 1 Processor -- start 2024-05-09T05:02:29.959 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.959+0000 7f48881e4640 1 -- start start 2024-05-09T05:02:29.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.960+0000 7f48881e4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f488015f3c0 0x7f4880162e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:29.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.960+0000 7f48881e4640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f48800b7fe0 0x7f48800b95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:29.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.960+0000 7f48881e4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48800742e0 con 0x7f48800b9b30 2024-05-09T05:02:29.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.960+0000 7f48881e4640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f4880163740 con 0x7f48800b7fe0 2024-05-09T05:02:29.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.960+0000 7f48881e4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4880163a20 con 0x7f488015f3c0 2024-05-09T05:02:29.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.960+0000 7f4886781640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f48800b9b30 0x7f4880071810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60676/0 (socket says 172.21.15.60:60676) 2024-05-09T05:02:29.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.960+0000 7f4886781640 1 -- 172.21.15.60:0/3848062860 learned_addr learned my addr 172.21.15.60:0/3848062860 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:02:29.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f4885f80640 1 --2- 172.21.15.60:0/3848062860 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f488015f3c0 0x7f4880162e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:29.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f488577f640 1 --2- 172.21.15.60:0/3848062860 >> v2:172.21.15.114:3300/0 conn(0x7f48800b7fe0 0x7f48800b95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:29.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f4884f7e640 1 -- 172.21.15.60:0/3848062860 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1497833722 0 0) 0x7f48800742e0 con 0x7f48800b9b30 2024-05-09T05:02:29.963 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f4884f7e640 1 -- 172.21.15.60:0/3848062860 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4868003520 con 0x7f48800b9b30 2024-05-09T05:02:29.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f4884f7e640 1 -- 172.21.15.60:0/3848062860 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4138077601 0 0) 0x7f4868003520 con 0x7f48800b9b30 2024-05-09T05:02:29.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f4884f7e640 1 -- 172.21.15.60:0/3848062860 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f488015f3c0 msgr2=0x7f4880162e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:29.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f4884f7e640 1 --2- 172.21.15.60:0/3848062860 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f488015f3c0 0x7f4880162e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:29.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f4884f7e640 1 -- 172.21.15.60:0/3848062860 >> v2:172.21.15.114:3300/0 conn(0x7f48800b7fe0 msgr2=0x7f48800b95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:29.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f4884f7e640 1 --2- 172.21.15.60:0/3848062860 >> v2:172.21.15.114:3300/0 conn(0x7f48800b7fe0 0x7f48800b95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:29.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f4884f7e640 1 -- 172.21.15.60:0/3848062860 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4880163d00 con 0x7f48800b9b30 2024-05-09T05:02:29.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f4884f7e640 1 -- 172.21.15.60:0/3848062860 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1503267047 0 0) 0x7f4870001a70 con 0x7f48800b9b30 2024-05-09T05:02:29.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f4884f7e640 1 -- 172.21.15.60:0/3848062860 >> v1:172.21.15.60:6789/0 conn(0x7f48800b9b30 legacy=0x7f4880071810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:29.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f4884f7e640 1 --2- 172.21.15.60:0/3848062860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4868003a00 0x7f4868003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:29.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f4884f7e640 1 --2- 172.21.15.60:0/3848062860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4868004800 0x7f4868004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:29.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f4884f7e640 1 --2- 172.21.15.60:0/3848062860 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f488015f3c0 0x7f48680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:29.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f4884f7e640 1 -- 172.21.15.60:0/3848062860 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f4880163740 con 0x7f4868004800 2024-05-09T05:02:29.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f4884f7e640 1 -- 172.21.15.60:0/3848062860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4880163a20 con 0x7f4868003a00 2024-05-09T05:02:29.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f4884f7e640 1 -- 172.21.15.60:0/3848062860 --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f486800b9c0 con 0x7f488015f3c0 2024-05-09T05:02:29.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f488577f640 1 --2- 172.21.15.60:0/3848062860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4868004800 0x7f4868004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:29.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f4885f80640 1 --2- 172.21.15.60:0/3848062860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4868003a00 0x7f4868003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:29.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f4886781640 1 --2- 172.21.15.60:0/3848062860 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f488015f3c0 0x7f48680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:29.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f488577f640 1 -- 172.21.15.60:0/3848062860 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f488015f3c0 msgr2=0x7f48680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:29.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f488577f640 1 --2- 172.21.15.60:0/3848062860 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f488015f3c0 0x7f48680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:29.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f488577f640 1 -- 172.21.15.60:0/3848062860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4868003a00 msgr2=0x7f4868003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:29.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f488577f640 1 --2- 172.21.15.60:0/3848062860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4868003a00 0x7f4868003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:29.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.961+0000 7f488577f640 1 -- 172.21.15.60:0/3848062860 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4868003520 con 0x7f4868004800 2024-05-09T05:02:29.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.962+0000 7f488577f640 1 --2- 172.21.15.60:0/3848062860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4868004800 0x7f4868004c80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4874009ac0 tx=0x7f4868005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa78a12f0180cac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:29.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.962+0000 7f4884f7e640 1 -- 172.21.15.60:0/3848062860 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4874036070 con 0x7f4868004800 2024-05-09T05:02:29.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.962+0000 7f4884f7e640 1 -- 172.21.15.60:0/3848062860 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4874031400 con 0x7f4868004800 2024-05-09T05:02:29.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.962+0000 7f4884f7e640 1 -- 172.21.15.60:0/3848062860 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4874009690 con 0x7f4868004800 2024-05-09T05:02:29.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.963+0000 7f48881e4640 1 -- 172.21.15.60:0/3848062860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4868004800 msgr2=0x7f4868004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:29.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.963+0000 7f48881e4640 1 --2- 172.21.15.60:0/3848062860 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4868004800 0x7f4868004c80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4874009ac0 tx=0x7f4868005830 comp rx=0 tx=0).stop 2024-05-09T05:02:29.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.963+0000 7f4886781640 1 -- 172.21.15.60:0/3848062860 reap_dead start 2024-05-09T05:02:29.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.963+0000 7f48881e4640 1 -- 172.21.15.60:0/3848062860 shutdown_connections 2024-05-09T05:02:29.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.963+0000 7f48881e4640 1 -- 172.21.15.60:0/3848062860 >> 172.21.15.60:0/3848062860 conn(0x7f488006cae0 msgr2=0x7f48800d53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:29.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.963+0000 7f48881e4640 1 -- 172.21.15.60:0/3848062860 shutdown_connections 2024-05-09T05:02:29.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.963+0000 7f48881e4640 1 -- 172.21.15.60:0/3848062860 wait complete. 2024-05-09T05:02:29.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.964+0000 7f48881e4640 1 Processor -- start 2024-05-09T05:02:29.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.964+0000 7f48881e4640 1 -- start start 2024-05-09T05:02:29.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.965+0000 7f48881e4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4868004800 0x7f4880158e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:29.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.965+0000 7f48881e4640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f488014fb80 0x7f488014ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:29.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.965+0000 7f48881e4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4880152b40 0x7f4880152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:29.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.965+0000 7f48881e4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f48800d1dc0 con 0x7f4868004800 2024-05-09T05:02:29.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.965+0000 7f48881e4640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4880072db0 con 0x7f4880152b40 2024-05-09T05:02:29.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.965+0000 7f48881e4640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f4880162e40 con 0x7f488014fb80 2024-05-09T05:02:29.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.965+0000 7f4885f80640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4868004800 0x7f4880158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:29.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.965+0000 7f4885f80640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4868004800 0x7f4880158e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59228/0 (socket says 172.21.15.60:59228) 2024-05-09T05:02:29.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.965+0000 7f4885f80640 1 -- 172.21.15.60:0/192795018 learned_addr learned my addr 172.21.15.60:0/192795018 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:29.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.965+0000 7f488577f640 1 --2- 172.21.15.60:0/192795018 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f488014fb80 0x7f488014ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:29.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.965+0000 7f4886781640 1 --2- 172.21.15.60:0/192795018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4880152b40 0x7f4880152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:29.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.965+0000 7f4885f80640 1 -- 172.21.15.60:0/192795018 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f488014fb80 msgr2=0x7f488014ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:29.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.965+0000 7f4885f80640 1 --2- 172.21.15.60:0/192795018 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f488014fb80 0x7f488014ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:29.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.965+0000 7f4885f80640 1 -- 172.21.15.60:0/192795018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4880152b40 msgr2=0x7f4880152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:29.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.965+0000 7f4885f80640 1 --2- 172.21.15.60:0/192795018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4880152b40 0x7f4880152fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:29.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.966+0000 7f4885f80640 1 -- 172.21.15.60:0/192795018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4880153720 con 0x7f4868004800 2024-05-09T05:02:29.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.966+0000 7f4885f80640 1 --2- 172.21.15.60:0/192795018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4868004800 0x7f4880158e40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f48780118e0 tx=0x7f4878011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:29.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.966+0000 7f487e7fc640 1 -- 172.21.15.60:0/192795018 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487800fa30 con 0x7f4868004800 2024-05-09T05:02:29.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.966+0000 7f487e7fc640 1 -- 172.21.15.60:0/192795018 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f487800fbd0 con 0x7f4868004800 2024-05-09T05:02:29.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.966+0000 7f48881e4640 1 -- 172.21.15.60:0/192795018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f488015f9b0 con 0x7f4868004800 2024-05-09T05:02:29.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.966+0000 7f487e7fc640 1 -- 172.21.15.60:0/192795018 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4878014440 con 0x7f4868004800 2024-05-09T05:02:29.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.966+0000 7f48881e4640 1 -- 172.21.15.60:0/192795018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f488015ff70 con 0x7f4868004800 2024-05-09T05:02:29.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.967+0000 7f48881e4640 1 -- 172.21.15.60:0/192795018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48800704c0 con 0x7f4868004800 2024-05-09T05:02:29.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.968+0000 7f487e7fc640 1 -- 172.21.15.60:0/192795018 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4878012040 con 0x7f4868004800 2024-05-09T05:02:29.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.968+0000 7f487e7fc640 1 --2- 172.21.15.60:0/192795018 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f486c0762d0 0x7f486c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:29.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.968+0000 7f487e7fc640 1 -- 172.21.15.60:0/192795018 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7f48780978b0 con 0x7f4868004800 2024-05-09T05:02:29.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.968+0000 7f488577f640 1 --2- 172.21.15.60:0/192795018 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f486c0762d0 0x7f486c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:29.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.969+0000 7f488577f640 1 --2- 172.21.15.60:0/192795018 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f486c0762d0 0x7f486c078790 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4874033000 tx=0x7f4874008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:29.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:29.972+0000 7f487e7fc640 1 -- 172.21.15.60:0/192795018 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f48800704c0 con 0x7f4868004800 2024-05-09T05:02:30.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.164+0000 7f48881e4640 1 -- 172.21.15.60:0/192795018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f4880163270 con 0x7f4868004800 2024-05-09T05:02:30.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.167+0000 7f487e7fc640 1 -- 172.21.15.60:0/192795018 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v17)=0 v17) ==== 123+0+0 (secure 0 0 0) 0x7f4880163270 con 0x7f4868004800 2024-05-09T05:02:30.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.168+0000 7f487e7fc640 1 -- 172.21.15.60:0/192795018 <== mon.0 v2:172.21.15.60:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4878060790 con 0x7f4868004800 2024-05-09T05:02:30.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.170+0000 7f48881e4640 1 -- 172.21.15.60:0/192795018 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f486c0762d0 msgr2=0x7f486c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:30.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.170+0000 7f48881e4640 1 --2- 172.21.15.60:0/192795018 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f486c0762d0 0x7f486c078790 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4874033000 tx=0x7f4874008030 comp rx=0 tx=0).stop 2024-05-09T05:02:30.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.170+0000 7f48881e4640 1 -- 172.21.15.60:0/192795018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4868004800 msgr2=0x7f4880158e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:30.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.170+0000 7f48881e4640 1 --2- 172.21.15.60:0/192795018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4868004800 0x7f4880158e40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f48780118e0 tx=0x7f4878011ce0 comp rx=0 tx=0).stop 2024-05-09T05:02:30.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.170+0000 7f48881e4640 1 -- 172.21.15.60:0/192795018 shutdown_connections 2024-05-09T05:02:30.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.170+0000 7f48881e4640 1 --2- 172.21.15.60:0/192795018 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f486c0762d0 0x7f486c078790 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:30.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.170+0000 7f48881e4640 1 --2- 172.21.15.60:0/192795018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4880152b40 0x7f4880152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:30.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.170+0000 7f48881e4640 1 --2- 172.21.15.60:0/192795018 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f488014fb80 0x7f488014ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:30.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.170+0000 7f48881e4640 1 --2- 172.21.15.60:0/192795018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4868004800 0x7f4880158e40 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:30.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.170+0000 7f48881e4640 1 -- 172.21.15.60:0/192795018 >> 172.21.15.60:0/192795018 conn(0x7f488006cae0 msgr2=0x7f488006fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:30.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.170+0000 7f48881e4640 1 -- 172.21.15.60:0/192795018 shutdown_connections 2024-05-09T05:02:30.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.170+0000 7f48881e4640 1 -- 172.21.15.60:0/192795018 wait complete. 2024-05-09T05:02:30.180 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T05:02:30.181 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:02:30.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.285+0000 7f6cfc22a640 1 Processor -- start 2024-05-09T05:02:30.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.285+0000 7f6cfc22a640 1 -- start start 2024-05-09T05:02:30.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.286+0000 7f6cfc22a640 1 --2- >> v2:172.21.15.114:3300/0 conn(0x7f6cf4073fb0 0x7f6cf40b4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:30.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.286+0000 7f6cfc22a640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6cf40b4b60 0x7f6cf4072140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:30.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.286+0000 7f6cfc22a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf415f7a0 con 0x7f6cf415f3c0 2024-05-09T05:02:30.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.286+0000 7f6cfc22a640 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 -- 0x7f6cf4071cb0 con 0x7f6cf40b4b60 2024-05-09T05:02:30.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.286+0000 7f6cfc22a640 1 -- --> v2:172.21.15.114:3300/0 -- mon_getmap magic: 0 -- 0x7f6cf40b4f90 con 0x7f6cf4073fb0 2024-05-09T05:02:30.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.286+0000 7f6cf9fc6640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6cf415f3c0 0x7f6cf409ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60684/0 (socket says 172.21.15.60:60684) 2024-05-09T05:02:30.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.286+0000 7f6cf9fc6640 1 -- 172.21.15.60:0/930109824 learned_addr learned my addr 172.21.15.60:0/930109824 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-05-09T05:02:30.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.286+0000 7f6cfa7c7640 1 --2- 172.21.15.60:0/930109824 >> v2:172.21.15.60:3300/0 conn(0x7f6cf40b4b60 0x7f6cf4072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:30.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.286+0000 7f6cf97c5640 1 --2- 172.21.15.60:0/930109824 >> v2:172.21.15.114:3300/0 conn(0x7f6cf4073fb0 0x7f6cf40b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:30.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.286+0000 7f6cf8fc4640 1 -- 172.21.15.60:0/930109824 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4264933782 0 0) 0x7f6cf415f7a0 con 0x7f6cf415f3c0 2024-05-09T05:02:30.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.286+0000 7f6cf8fc4640 1 -- 172.21.15.60:0/930109824 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ce0003520 con 0x7f6cf415f3c0 2024-05-09T05:02:30.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.286+0000 7f6cfa7c7640 1 -- 172.21.15.60:0/930109824 >> v2:172.21.15.114:3300/0 conn(0x7f6cf4073fb0 msgr2=0x7f6cf40b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:02:30.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.286+0000 7f6cfa7c7640 1 --2- 172.21.15.60:0/930109824 >> v2:172.21.15.114:3300/0 conn(0x7f6cf4073fb0 0x7f6cf40b4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:30.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.286+0000 7f6cfa7c7640 1 -- 172.21.15.60:0/930109824 >> v1:172.21.15.60:6789/0 conn(0x7f6cf415f3c0 legacy=0x7f6cf409ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:30.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.286+0000 7f6cfa7c7640 1 -- 172.21.15.60:0/930109824 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf409f480 con 0x7f6cf40b4b60 2024-05-09T05:02:30.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.287+0000 7f6cfa7c7640 1 --2- 172.21.15.60:0/930109824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf40b4b60 0x7f6cf4072140 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4070080 tx=0x7f6cec02a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc1bb4f30d9e3a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:30.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.287+0000 7f6cf8fc4640 1 -- 172.21.15.60:0/930109824 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec007740 con 0x7f6cf40b4b60 2024-05-09T05:02:30.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.287+0000 7f6cf8fc4640 1 -- 172.21.15.60:0/930109824 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f6cec0078e0 con 0x7f6cf40b4b60 2024-05-09T05:02:30.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.287+0000 7f6cf8fc4640 1 -- 172.21.15.60:0/930109824 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec007c90 con 0x7f6cf40b4b60 2024-05-09T05:02:30.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.288+0000 7f6cfc22a640 1 -- 172.21.15.60:0/930109824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf40b4b60 msgr2=0x7f6cf4072140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:30.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.288+0000 7f6cfc22a640 1 --2- 172.21.15.60:0/930109824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf40b4b60 0x7f6cf4072140 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4070080 tx=0x7f6cec02a5f0 comp rx=0 tx=0).stop 2024-05-09T05:02:30.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.288+0000 7f6cfc22a640 1 -- 172.21.15.60:0/930109824 shutdown_connections 2024-05-09T05:02:30.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.288+0000 7f6cfc22a640 1 --2- 172.21.15.60:0/930109824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf40b4b60 0x7f6cf4072140 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:30.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.288+0000 7f6cfc22a640 1 --2- 172.21.15.60:0/930109824 >> v2:172.21.15.114:3300/0 conn(0x7f6cf4073fb0 0x7f6cf40b4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:30.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.288+0000 7f6cfc22a640 1 -- 172.21.15.60:0/930109824 >> 172.21.15.60:0/930109824 conn(0x7f6cf40d1470 msgr2=0x7f6cf40d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:30.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.288+0000 7f6cfc22a640 1 -- 172.21.15.60:0/930109824 shutdown_connections 2024-05-09T05:02:30.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.288+0000 7f6cfc22a640 1 -- 172.21.15.60:0/930109824 wait complete. 2024-05-09T05:02:30.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.289+0000 7f6cfc22a640 1 Processor -- start 2024-05-09T05:02:30.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.289+0000 7f6cfc22a640 1 -- start start 2024-05-09T05:02:30.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.290+0000 7f6cfc22a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf4073fb0 0x7f6cf415d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:30.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.290+0000 7f6cfc22a640 1 --2- >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6cf40b4b60 0x7f6cf40c9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:30.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.290+0000 7f6cfc22a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf415f3c0 0x7f6cf40cb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:30.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.290+0000 7f6cfc22a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf4072680 con 0x7f6cf415f3c0 2024-05-09T05:02:30.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.290+0000 7f6cfc22a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf4071940 con 0x7f6cf4073fb0 2024-05-09T05:02:30.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.290+0000 7f6cfc22a640 1 -- --> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] -- mon_getmap magic: 0 -- 0x7f6cf4072800 con 0x7f6cf40b4b60 2024-05-09T05:02:30.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.290+0000 7f6cfa7c7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf415f3c0 0x7f6cf40cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:30.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.290+0000 7f6cfa7c7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf415f3c0 0x7f6cf40cb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59242/0 (socket says 172.21.15.60:59242) 2024-05-09T05:02:30.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.290+0000 7f6cfa7c7640 1 -- 172.21.15.60:0/3616861610 learned_addr learned my addr 172.21.15.60:0/3616861610 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-05-09T05:02:30.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.290+0000 7f6cf97c5640 1 --2- 172.21.15.60:0/3616861610 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6cf40b4b60 0x7f6cf40c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:30.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.290+0000 7f6cf9fc6640 1 --2- 172.21.15.60:0/3616861610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf4073fb0 0x7f6cf415d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:30.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.290+0000 7f6cfa7c7640 1 -- 172.21.15.60:0/3616861610 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6cf40b4b60 msgr2=0x7f6cf40c9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:30.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.290+0000 7f6cfa7c7640 1 --2- 172.21.15.60:0/3616861610 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6cf40b4b60 0x7f6cf40c9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:30.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.290+0000 7f6cfa7c7640 1 -- 172.21.15.60:0/3616861610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf4073fb0 msgr2=0x7f6cf415d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:30.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.290+0000 7f6cfa7c7640 1 --2- 172.21.15.60:0/3616861610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf4073fb0 0x7f6cf415d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:30.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.290+0000 7f6cfa7c7640 1 -- 172.21.15.60:0/3616861610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf40c9d10 con 0x7f6cf415f3c0 2024-05-09T05:02:30.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.290+0000 7f6cf97c5640 1 --2- 172.21.15.60:0/3616861610 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6cf40b4b60 0x7f6cf40c9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:02:30.295 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.290+0000 7f6cf9fc6640 1 --2- 172.21.15.60:0/3616861610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf4073fb0 0x7f6cf415d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:02:30.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.291+0000 7f6cfa7c7640 1 --2- 172.21.15.60:0/3616861610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf415f3c0 0x7f6cf40cb090 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f6cec02a630 tx=0x7f6cec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:30.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.291+0000 7f6cea7fc640 1 -- 172.21.15.60:0/3616861610 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec040070 con 0x7f6cf415f3c0 2024-05-09T05:02:30.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.291+0000 7f6cea7fc640 1 -- 172.21.15.60:0/3616861610 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f6cec031c40 con 0x7f6cf415f3c0 2024-05-09T05:02:30.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.291+0000 7f6cfc22a640 1 -- 172.21.15.60:0/3616861610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf40c9ff0 con 0x7f6cf415f3c0 2024-05-09T05:02:30.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.291+0000 7f6cea7fc640 1 -- 172.21.15.60:0/3616861610 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec009450 con 0x7f6cf415f3c0 2024-05-09T05:02:30.296 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.291+0000 7f6cfc22a640 1 -- 172.21.15.60:0/3616861610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf40ca5b0 con 0x7f6cf415f3c0 2024-05-09T05:02:30.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.292+0000 7f6cfc22a640 1 -- 172.21.15.60:0/3616861610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf409dd60 con 0x7f6cf415f3c0 2024-05-09T05:02:30.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.292+0000 7f6cea7fc640 1 -- 172.21.15.60:0/3616861610 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6cec039040 con 0x7f6cf415f3c0 2024-05-09T05:02:30.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.293+0000 7f6cea7fc640 1 --2- 172.21.15.60:0/3616861610 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6cd80762d0 0x7f6cd8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:02:30.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.293+0000 7f6cea7fc640 1 -- 172.21.15.60:0/3616861610 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5335+0+0 (secure 0 0 0) 0x7f6cec0b87c0 con 0x7f6cf415f3c0 2024-05-09T05:02:30.297 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.293+0000 7f6cf9fc6640 1 --2- 172.21.15.60:0/3616861610 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6cd80762d0 0x7f6cd8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:02:30.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.294+0000 7f6cf9fc6640 1 --2- 172.21.15.60:0/3616861610 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6cd80762d0 0x7f6cd8078790 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0007fd0 tx=0x7f6cf0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:02:30.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.297+0000 7f6cea7fc640 1 -- 172.21.15.60:0/3616861610 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6cf409dd60 con 0x7f6cf415f3c0 2024-05-09T05:02:30.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.526+0000 7f6cfc22a640 1 -- 172.21.15.60:0/3616861610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6cf4051040 con 0x7f6cf415f3c0 2024-05-09T05:02:30.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.527+0000 7f6cea7fc640 1 -- 172.21.15.60:0/3616861610 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+3875 (secure 0 0 0) 0x7f6cec082200 con 0x7f6cf415f3c0 2024-05-09T05:02:30.527 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-05-09T05:02:30.527 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":31,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6834/3604145570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3604145570},{"type":"v1","addr":"172.21.15.60:6834","nonce":3604145570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5161,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6836/3949194455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3949194455},{"type":"v1","addr":"172.21.15.114:6836","nonce":3949194455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5164,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/3288630382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6835","nonce":3288630382},{"type":"v1","addr":"172.21.15.114:6837","nonce":3288630382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5673,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/4293575371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6833","nonce":4293575371},{"type":"v1","addr":"172.21.15.60:6835","nonce":4293575371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[]} 2024-05-09T05:02:30.527 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 31 2024-05-09T05:02:30.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.528+0000 7f6cfc22a640 1 -- 172.21.15.60:0/3616861610 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6cd80762d0 msgr2=0x7f6cd8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:30.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.528+0000 7f6cfc22a640 1 --2- 172.21.15.60:0/3616861610 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6cd80762d0 0x7f6cd8078790 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0007fd0 tx=0x7f6cf0009040 comp rx=0 tx=0).stop 2024-05-09T05:02:30.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.528+0000 7f6cfc22a640 1 -- 172.21.15.60:0/3616861610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf415f3c0 msgr2=0x7f6cf40cb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:02:30.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.528+0000 7f6cfc22a640 1 --2- 172.21.15.60:0/3616861610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf415f3c0 0x7f6cf40cb090 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f6cec02a630 tx=0x7f6cec004a40 comp rx=0 tx=0).stop 2024-05-09T05:02:30.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.529+0000 7f6cfc22a640 1 -- 172.21.15.60:0/3616861610 shutdown_connections 2024-05-09T05:02:30.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.529+0000 7f6cfc22a640 1 --2- 172.21.15.60:0/3616861610 >> [v2:172.21.15.114:6832/4075088736,v1:172.21.15.114:6833/4075088736] conn(0x7f6cd80762d0 0x7f6cd8078790 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:30.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.529+0000 7f6cfc22a640 1 --2- 172.21.15.60:0/3616861610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6cf415f3c0 0x7f6cf40cb090 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:30.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.529+0000 7f6cfc22a640 1 --2- 172.21.15.60:0/3616861610 >> [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] conn(0x7f6cf40b4b60 0x7f6cf40c9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:30.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.529+0000 7f6cfc22a640 1 --2- 172.21.15.60:0/3616861610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cf4073fb0 0x7f6cf415d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:02:30.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.529+0000 7f6cfc22a640 1 -- 172.21.15.60:0/3616861610 >> 172.21.15.60:0/3616861610 conn(0x7f6cf40d1470 msgr2=0x7f6cf409a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:02:30.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.529+0000 7f6cfc22a640 1 -- 172.21.15.60:0/3616861610 shutdown_connections 2024-05-09T05:02:30.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-05-09T05:02:30.529+0000 7f6cfc22a640 1 -- 172.21.15.60:0/3616861610 wait complete. 2024-05-09T05:02:30.539 DEBUG:tasks.cephfs.filesystem:already dead... 2024-05-09T05:02:30.539 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-09T05:02:30.540 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-09T05:02:30.540 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:02:30.548 INFO:tasks.ceph.mds.a:Stopped 2024-05-09T05:02:30.549 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-09T05:02:30.549 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:02:30.599 INFO:tasks.ceph.mds.c:Stopped 2024-05-09T05:02:30.599 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-09T05:02:30.599 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:02:30.611 INFO:tasks.ceph.mds.b:Stopped 2024-05-09T05:02:30.611 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-09T05:02:30.612 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:02:30.663 INFO:tasks.ceph.mds.d:Stopped 2024-05-09T05:02:30.663 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-09T05:02:30.664 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-05-09T05:02:30.664 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:02:30.700 INFO:tasks.ceph.osd.0:Stopped 2024-05-09T05:02:30.701 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-05-09T05:02:30.701 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:02:30.772 INFO:tasks.ceph.osd.1:Stopped 2024-05-09T05:02:30.773 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-05-09T05:02:30.773 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:02:30.844 INFO:tasks.ceph.osd.2:Stopped 2024-05-09T05:02:30.845 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-05-09T05:02:30.845 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:02:30.916 INFO:tasks.ceph.osd.3:Stopped 2024-05-09T05:02:30.917 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-05-09T05:02:30.917 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:02:30.950 INFO:tasks.ceph.osd.4:Stopped 2024-05-09T05:02:30.950 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-05-09T05:02:30.950 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:02:31.026 INFO:tasks.ceph.osd.5:Stopped 2024-05-09T05:02:31.027 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-05-09T05:02:31.027 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:02:31.103 INFO:tasks.ceph.osd.6:Stopped 2024-05-09T05:02:31.104 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-05-09T05:02:31.104 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:02:31.179 INFO:tasks.ceph.osd.7:Stopped 2024-05-09T05:02:31.179 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-09T05:02:31.180 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-05-09T05:02:31.180 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:02:31.206 INFO:tasks.ceph.mgr.y:Stopped 2024-05-09T05:02:31.206 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-05-09T05:02:31.206 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:02:31.249 INFO:tasks.ceph.mgr.x:Stopped 2024-05-09T05:02:31.249 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-09T05:02:31.249 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-09T05:02:31.249 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:02:31.260 INFO:tasks.ceph.mon.a:Stopped 2024-05-09T05:02:31.261 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-09T05:02:31.261 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:02:31.302 INFO:tasks.ceph.mon.b:Stopped 2024-05-09T05:02:31.302 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-09T05:02:31.302 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:02:31.355 INFO:tasks.ceph.mon.c:Stopped 2024-05-09T05:02:31.355 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-09T05:02:31.356 DEBUG:teuthology.orchestra.run.smithi060:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | head -n 1 2024-05-09T05:02:31.395 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi060.front.sepia.ceph.com 2024-05-09T05:02:31.396 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-05-09T05:02:31.631 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi060.front.sepia.ceph.com 2024-05-09T05:02:31.631 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-05-09T05:02:31.726 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi060.front.sepia.ceph.com 2024-05-09T05:02:31.727 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-05-09T05:02:31.768 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi060.front.sepia.ceph.com 2024-05-09T05:02:31.768 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-05-09T05:02:31.877 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T05:02:31.877 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-05-09T05:02:32.017 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T05:02:32.018 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-05-09T05:02:32.113 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T05:02:32.113 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-05-09T05:02:32.170 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T05:02:32.171 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-05-09T05:02:32.228 INFO:tasks.ceph:Archiving mon data... 2024-05-09T05:02:32.229 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi/7699557/data/mon.a.tgz 2024-05-09T05:02:32.229 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-05-09T05:02:32.246 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.Em5AvbfVst 2024-05-09T05:02:32.246 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Em5AvbfVst 2024-05-09T05:02:32.389 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.Em5AvbfVst 2024-05-09T05:02:32.474 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.Em5AvbfVst is 350KB 2024-05-09T05:02:32.572 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.Em5AvbfVst 2024-05-09T05:02:32.587 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi/7699557/data/mon.b.tgz 2024-05-09T05:02:32.588 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-05-09T05:02:32.606 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.0kO4X7IIZI 2024-05-09T05:02:32.607 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.0kO4X7IIZI 2024-05-09T05:02:32.799 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.0kO4X7IIZI 2024-05-09T05:02:32.885 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.0kO4X7IIZI is 467KB 2024-05-09T05:02:32.982 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.0kO4X7IIZI 2024-05-09T05:02:32.998 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi/7699557/data/mon.c.tgz 2024-05-09T05:02:32.999 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-05-09T05:02:33.053 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.otUGnVrJ03 2024-05-09T05:02:33.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.otUGnVrJ03 2024-05-09T05:02:33.204 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.otUGnVrJ03 2024-05-09T05:02:33.288 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.otUGnVrJ03 is 467KB 2024-05-09T05:02:33.387 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.otUGnVrJ03 2024-05-09T05:02:33.403 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-09T05:02:33.403 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-09T05:02:33.407 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-09T05:02:33.494 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-09T05:02:33.523 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-09T05:02:33.523 INFO:tasks.ceph:Archiving crash dumps... 2024-05-09T05:02:33.525 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi/7699557/remote/smithi060/crash 2024-05-09T05:02:33.526 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-09T05:02:33.565 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi/7699557/remote/smithi114/crash 2024-05-09T05:02:33.566 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-09T05:02:33.591 INFO:tasks.ceph:Shutting down logrotate 2024-05-09T05:02:33.591 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:02:33.607 DEBUG:teuthology.orchestra.run.smithi114:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:02:33.684 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-05-09T05:02:33.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-05-09T05:02:33.738 INFO:tasks.ceph:Compressing logs... 2024-05-09T05:02:33.739 DEBUG:teuthology.orchestra.run.smithi060:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T05:02:33.757 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T05:02:33.782 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52271.log 2024-05-09T05:02:33.822 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.42434.log 2024-05-09T05:02:33.822 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-09T05:02:33.823 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-09T05:02:33.823 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.3.42434.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.42434.log.gz 2024-05-09T05:02:33.823 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35712.log 2024-05-09T05:02:33.823 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-05-09T05:02:33.823 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35712.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-05-09T05:02:33.823 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.7.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35712.log.gz 2024-05-09T05:02:33.823 INFO:teuthology.orchestra.run.smithi114.stderr: 92.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-09T05:02:33.823 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-05-09T05:02:33.824 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-05-09T05:02:33.824 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42037.log 2024-05-09T05:02:33.824 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35628.log 2024-05-09T05:02:33.824 INFO:teuthology.orchestra.run.smithi114.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-09T05:02:33.824 INFO:teuthology.orchestra.run.smithi114.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mds.b.log.gz/var/log/ceph/ceph-client.1.42037.log: 2024-05-09T05:02:33.824 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-05-09T05:02:33.824 INFO:teuthology.orchestra.run.smithi114.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.42037.log.gz 2024-05-09T05:02:33.824 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35628.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-05-09T05:02:33.825 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mgr.x.log: 87.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.35628.log.gz -5 2024-05-09T05:02:33.825 INFO:teuthology.orchestra.run.smithi114.stderr: --verbose -- /var/log/ceph/ceph-osd.6.log 2024-05-09T05:02:33.825 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35588.log 2024-05-09T05:02:33.825 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35879.log 2024-05-09T05:02:33.825 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35668.log 2024-05-09T05:02:33.825 INFO:teuthology.orchestra.run.smithi114.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35588.log.gz 2024-05-09T05:02:33.825 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35839.log 2024-05-09T05:02:33.825 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35668.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35879.log.gz 2024-05-09T05:02:33.826 INFO:teuthology.orchestra.run.smithi114.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35668.log.gz 2024-05-09T05:02:33.826 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35759.log 2024-05-09T05:02:33.826 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35799.log 2024-05-09T05:02:33.826 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35839.log: /var/log/ceph/ceph-client.admin.35759.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35839.log.gz 2024-05-09T05:02:33.826 INFO:teuthology.orchestra.run.smithi114.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35759.log.gz 2024-05-09T05:02:33.826 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-05-09T05:02:33.826 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-05-09T05:02:33.826 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35799.log: /var/log/ceph/ceph-mds.d.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35799.log.gz 2024-05-09T05:02:33.827 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mon.b.log: 92.2% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-05-09T05:02:33.856 INFO:teuthology.orchestra.run.smithi114.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-05-09T05:02:33.918 INFO:teuthology.orchestra.run.smithi114.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-05-09T05:02:33.918 INFO:teuthology.orchestra.run.smithi114.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-05-09T05:02:33.921 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40939.log 2024-05-09T05:02:33.921 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48927.log 2024-05-09T05:02:33.921 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40939.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52271.log.gz 2024-05-09T05:02:33.921 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43227.log 2024-05-09T05:02:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48927.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40939.log.gz 2024-05-09T05:02:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48487.log 2024-05-09T05:02:33.922 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.48927.log.gz 2024-05-09T05:02:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45030.log 2024-05-09T05:02:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35344.log 2024-05-09T05:02:33.922 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43227.log.gz 2024-05-09T05:02:33.922 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40634.log 2024-05-09T05:02:33.922 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.48487.log.gz 2024-05-09T05:02:33.923 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45030.log.gz 2024-05-09T05:02:33.923 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47764.log 2024-05-09T05:02:33.923 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40634.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35344.log.gz 2024-05-09T05:02:33.923 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45794.log 2024-05-09T05:02:33.923 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40634.log.gz 2024-05-09T05:02:33.923 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46061.log 2024-05-09T05:02:33.923 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45794.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47764.log.gz 2024-05-09T05:02:33.923 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42546.log 2024-05-09T05:02:33.924 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45794.log.gz 2024-05-09T05:02:33.924 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46805.log 2024-05-09T05:02:33.924 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46061.log.gz 2024-05-09T05:02:33.924 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44820.log 2024-05-09T05:02:33.924 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42546.log.gz 2024-05-09T05:02:33.924 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51549.log 2024-05-09T05:02:33.924 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46805.log.gz 2024-05-09T05:02:33.925 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46633.log 2024-05-09T05:02:33.925 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44820.log.gz 2024-05-09T05:02:33.925 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47610.log 2024-05-09T05:02:33.925 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51549.log.gz 2024-05-09T05:02:33.925 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43470.log 2024-05-09T05:02:33.925 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46633.log.gz 2024-05-09T05:02:33.925 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52355.log 2024-05-09T05:02:33.925 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47610.log.gz 2024-05-09T05:02:33.925 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42888.log 2024-05-09T05:02:33.926 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43470.log.gz 2024-05-09T05:02:33.926 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49290.log 2024-05-09T05:02:33.926 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52355.log.gz 2024-05-09T05:02:33.926 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53497.log 2024-05-09T05:02:33.926 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42888.log.gz 2024-05-09T05:02:33.926 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46763.log 2024-05-09T05:02:33.926 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46452.log 2024-05-09T05:02:33.926 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49290.log.gz 2024-05-09T05:02:33.927 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46763.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53497.log.gz 2024-05-09T05:02:33.927 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42378.log 2024-05-09T05:02:33.927 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46452.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46763.log.gz 2024-05-09T05:02:33.927 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46353.log 2024-05-09T05:02:33.927 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46452.log.gz 2024-05-09T05:02:33.927 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49897.log 2024-05-09T05:02:33.927 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42378.log.gz 2024-05-09T05:02:33.927 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46353.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-09T05:02:33.928 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46353.log.gz 2024-05-09T05:02:33.928 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53457.log 2024-05-09T05:02:33.928 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49897.log.gz 2024-05-09T05:02:33.928 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43512.log 2024-05-09T05:02:33.928 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52965.log 2024-05-09T05:02:33.928 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53457.log.gz 2024-05-09T05:02:33.928 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49332.log 2024-05-09T05:02:33.928 INFO:teuthology.orchestra.run.smithi060.stderr: 85.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-09T05:02:33.929 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43512.log.gz 2024-05-09T05:02:33.929 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52629.log 2024-05-09T05:02:33.929 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49332.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52965.log.gz 2024-05-09T05:02:33.929 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48689.log 2024-05-09T05:02:33.929 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49332.log.gz 2024-05-09T05:02:33.929 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45225.log 2024-05-09T05:02:33.929 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51507.log 2024-05-09T05:02:33.929 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52629.log.gz 2024-05-09T05:02:33.930 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45225.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48689.log.gz 2024-05-09T05:02:33.930 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48132.log 2024-05-09T05:02:33.930 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51507.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45225.log.gz 2024-05-09T05:02:33.930 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-05-09T05:02:33.930 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48132.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51507.log.gz 2024-05-09T05:02:33.930 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41956.log 2024-05-09T05:02:33.930 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43428.log 2024-05-09T05:02:33.930 INFO:teuthology.orchestra.run.smithi060.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.48132.log.gz 2024-05-09T05:02:33.931 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49519.log 2024-05-09T05:02:33.931 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45303.log 2024-05-09T05:02:33.931 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49519.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43428.log.gz 2024-05-09T05:02:33.931 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45597.log 2024-05-09T05:02:33.931 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49519.log.gz 2024-05-09T05:02:33.931 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29327.log 2024-05-09T05:02:33.931 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45303.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.41956.log.gz 2024-05-09T05:02:33.931 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40855.log 2024-05-09T05:02:33.932 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45303.log.gz 2024-05-09T05:02:33.932 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph.tmp-client.admin.29327.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.45597.log.gz -5 0.0% 2024-05-09T05:02:33.932 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- -- replaced with /var/log/ceph/ceph.tmp-client.admin.29327.log.gz 2024-05-09T05:02:33.932 INFO:teuthology.orchestra.run.smithi060.stderr: /var/log/ceph/ceph-client.admin.49603.log 2024-05-09T05:02:33.932 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46847.log 2024-05-09T05:02:33.932 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40855.log.gz 2024-05-09T05:02:33.932 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45393.log 2024-05-09T05:02:33.932 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49603.log.gz 2024-05-09T05:02:33.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49374.log 2024-05-09T05:02:33.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45393.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46847.log.gz 2024-05-09T05:02:33.933 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51055.log 2024-05-09T05:02:33.933 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45393.log.gz 2024-05-09T05:02:33.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43554.log 2024-05-09T05:02:33.933 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49374.log.gz 2024-05-09T05:02:33.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.51055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49992.log 2024-05-09T05:02:33.933 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.51055.log.gz 2024-05-09T05:02:33.934 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50237.log 2024-05-09T05:02:33.934 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43554.log.gz 2024-05-09T05:02:33.934 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51220.log 2024-05-09T05:02:33.934 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49992.log.gz 2024-05-09T05:02:33.934 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50237.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-09T05:02:33.934 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51220.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50237.log.gz 2024-05-09T05:02:33.934 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43231.log 2024-05-09T05:02:33.935 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51220.log.gz 2024-05-09T05:02:33.935 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50549.log 2024-05-09T05:02:33.935 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49771.log 2024-05-09T05:02:33.935 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50549.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43231.log.gz 2024-05-09T05:02:33.935 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52797.log 2024-05-09T05:02:33.935 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50549.log.gz 2024-05-09T05:02:33.935 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.49771.log: -- /var/log/ceph/ceph-client.admin.53175.log 2024-05-09T05:02:33.935 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-09T05:02:33.936 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43722.log 2024-05-09T05:02:33.936 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49771.log.gz 2024-05-09T05:02:33.936 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53175.log: gzip -5 --verbose 88.3% -- -- replaced with /var/log/ceph/ceph-client.admin.52797.log.gz /var/log/ceph/ceph-client.admin.42895.log 2024-05-09T05:02:33.936 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-05-09T05:02:33.936 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53175.log.gz 2024-05-09T05:02:33.936 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43230.log 2024-05-09T05:02:33.936 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35608.log 2024-05-09T05:02:33.936 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43230.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.43722.log.gz 2024-05-09T05:02:33.936 INFO:teuthology.orchestra.run.smithi060.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42895.log.gz 2024-05-09T05:02:33.937 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46141.log 2024-05-09T05:02:33.937 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43230.log.gz 2024-05-09T05:02:33.937 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49561.log 2024-05-09T05:02:33.937 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35608.log.gz 2024-05-09T05:02:33.937 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48290.log 2024-05-09T05:02:33.937 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49561.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46141.log.gzgzip 2024-05-09T05:02:33.937 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43224.log 2024-05-09T05:02:33.937 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49561.log.gz 2024-05-09T05:02:33.938 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43232.log 2024-05-09T05:02:33.938 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48290.log.gz 2024-05-09T05:02:33.938 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42891.log 2024-05-09T05:02:33.938 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43232.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43224.log.gz 2024-05-09T05:02:33.938 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47721.log 2024-05-09T05:02:33.938 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43232.log.gz 2024-05-09T05:02:33.938 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48646.log 2024-05-09T05:02:33.938 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42891.log.gz 2024-05-09T05:02:33.939 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42892.log 2024-05-09T05:02:33.939 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48646.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47721.log.gz 2024-05-09T05:02:33.939 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45981.log 2024-05-09T05:02:33.939 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48646.log.gz 2024-05-09T05:02:33.939 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44340.log 2024-05-09T05:02:33.939 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42892.log.gz 2024-05-09T05:02:33.939 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41528.log 2024-05-09T05:02:33.939 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44340.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45981.log.gz 2024-05-09T05:02:33.940 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-05-09T05:02:33.940 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44340.log.gz 2024-05-09T05:02:33.940 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40983.log 2024-05-09T05:02:33.940 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41528.log.gz 2024-05-09T05:02:33.940 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46311.log 2024-05-09T05:02:33.940 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41124.log 2024-05-09T05:02:33.940 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40983.log.gz 2024-05-09T05:02:33.940 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42893.log 2024-05-09T05:02:33.941 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51832.log 2024-05-09T05:02:33.941 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46311.log.gz 2024-05-09T05:02:33.941 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41124.log: /var/log/ceph/ceph-client.admin.42893.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41124.log.gz 2024-05-09T05:02:33.941 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49645.log 2024-05-09T05:02:33.941 INFO:teuthology.orchestra.run.smithi060.stderr:gzip 87.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.42893.log.gz --verbose 2024-05-09T05:02:33.941 INFO:teuthology.orchestra.run.smithi060.stderr: -- /var/log/ceph/ceph-client.admin.49416.log 2024-05-09T05:02:33.941 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51832.log: /var/log/ceph/ceph-client.admin.49645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48434.log 2024-05-09T05:02:33.941 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51832.log.gz 2024-05-09T05:02:33.941 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43596.log 2024-05-09T05:02:33.942 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49416.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49645.log.gz 2024-05-09T05:02:33.942 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48434.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49416.log.gz 2024-05-09T05:02:33.942 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47930.log 2024-05-09T05:02:33.942 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45554.log 2024-05-09T05:02:33.942 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43596.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.48434.log.gz 2024-05-09T05:02:33.942 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47930.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43596.log.gz -5 2024-05-09T05:02:33.942 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51696.log 2024-05-09T05:02:33.942 INFO:teuthology.orchestra.run.smithi060.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.47930.log.gz 2024-05-09T05:02:33.943 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44904.log 2024-05-09T05:02:33.943 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45554.log: /var/log/ceph/ceph-client.admin.51696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52587.log 2024-05-09T05:02:33.943 INFO:teuthology.orchestra.run.smithi060.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45554.log.gz 2024-05-09T05:02:33.943 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51696.log.gz 2024-05-09T05:02:33.943 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47390.log 2024-05-09T05:02:33.943 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52587.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44904.log.gz 2024-05-09T05:02:33.943 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46931.log 2024-05-09T05:02:33.943 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52587.log.gz 2024-05-09T05:02:33.944 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52187.log 2024-05-09T05:02:33.944 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47390.log.gz 2024-05-09T05:02:33.944 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46931.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-05-09T05:02:33.944 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46931.log.gz 2024-05-09T05:02:33.944 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53343.log 2024-05-09T05:02:33.944 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52187.log: /var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49729.log 2024-05-09T05:02:33.944 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52187.log.gz 2024-05-09T05:02:33.944 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44205.log 2024-05-09T05:02:33.945 INFO:teuthology.orchestra.run.smithi060.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53343.log.gz 2024-05-09T05:02:33.945 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52923.log 2024-05-09T05:02:33.945 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.44205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.44205.log.gz 2024-05-09T05:02:33.945 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47554.log 2024-05-09T05:02:33.945 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49729.log.gz 2024-05-09T05:02:33.945 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51790.log 2024-05-09T05:02:33.945 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52923.log: /var/log/ceph/ceph-client.admin.47554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42336.log 2024-05-09T05:02:33.945 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52923.log.gz 2024-05-09T05:02:33.946 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51790.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-05-09T05:02:33.946 INFO:teuthology.orchestra.run.smithi060.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47554.log.gz 2024-05-09T05:02:33.946 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51790.log.gz 2024-05-09T05:02:33.946 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48874.log 2024-05-09T05:02:33.946 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-05-09T05:02:33.946 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42336.log.gz 2024-05-09T05:02:33.946 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.48874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48248.log 2024-05-09T05:02:33.946 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mgr.y.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48874.log.gz 2024-05-09T05:02:33.947 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42462.log 2024-05-09T05:02:33.947 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46494.log 2024-05-09T05:02:33.947 INFO:teuthology.orchestra.run.smithi060.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48248.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42216.log 2024-05-09T05:02:33.947 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-05-09T05:02:33.947 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42462.log: /var/log/ceph/ceph-client.admin.46494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42886.log 2024-05-09T05:02:33.947 INFO:teuthology.orchestra.run.smithi060.stderr: 89.2%/var/log/ceph/ceph-client.admin.42216.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42462.log.gz 2024-05-09T05:02:33.947 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46494.log.gz 2024-05-09T05:02:33.948 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42216.log.gz 2024-05-09T05:02:33.948 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41294.log 2024-05-09T05:02:33.948 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49052.log 2024-05-09T05:02:33.948 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48742.log 2024-05-09T05:02:33.948 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6%/var/log/ceph/ceph-client.admin.41294.log: -- replaced with /var/log/ceph/ceph-client.admin.42886.log.gz 2024-05-09T05:02:33.948 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49052.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41294.log.gz 2024-05-09T05:02:33.948 INFO:teuthology.orchestra.run.smithi060.stderr:gzip 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49052.log.gz 2024-05-09T05:02:33.949 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45512.log 2024-05-09T05:02:33.949 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48742.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-05-09T05:02:33.949 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% 92.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-05-09T05:02:33.949 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48742.log.gz 2024-05-09T05:02:33.949 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47983.log 2024-05-09T05:02:33.949 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45512.log.gz 2024-05-09T05:02:33.949 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46225.log 2024-05-09T05:02:33.950 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45072.log 2024-05-09T05:02:33.950 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52229.log 2024-05-09T05:02:33.950 INFO:teuthology.orchestra.run.smithi060.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.47983.log.gz 2024-05-09T05:02:33.950 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45072.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46225.log.gz 2024-05-09T05:02:33.950 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45072.log.gz 2024-05-09T05:02:33.950 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53133.log 2024-05-09T05:02:33.950 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43886.log 2024-05-09T05:02:33.950 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53133.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52229.log.gz 2024-05-09T05:02:33.951 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53385.log 2024-05-09T05:02:33.951 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53133.log.gz 2024-05-09T05:02:33.951 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45470.log 2024-05-09T05:02:33.951 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43886.log: /var/log/ceph/ceph-client.admin.53385.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43886.log.gz 2024-05-09T05:02:33.951 INFO:teuthology.orchestra.run.smithi060.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53385.log.gz 2024-05-09T05:02:33.951 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52713.log 2024-05-09T05:02:33.951 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50759.log 2024-05-09T05:02:33.952 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4%/var/log/ceph/ceph-client.admin.52713.log: -- replaced with /var/log/ceph/ceph-client.admin.45470.log.gz 2024-05-09T05:02:33.952 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42714.log 2024-05-09T05:02:33.952 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52713.log.gz 2024-05-09T05:02:33.952 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5/var/log/ceph/ceph-client.admin.50759.log: --verbose -- /var/log/ceph/ceph-client.admin.51958.log 2024-05-09T05:02:33.952 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42714.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50759.log.gz 2024-05-09T05:02:33.952 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50801.log 2024-05-09T05:02:33.952 INFO:teuthology.orchestra.run.smithi060.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42714.log.gz 2024-05-09T05:02:33.952 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49855.log 2024-05-09T05:02:33.953 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51958.log.gz 2024-05-09T05:02:33.953 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45897.log 2024-05-09T05:02:33.953 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49206.log 2024-05-09T05:02:33.953 INFO:teuthology.orchestra.run.smithi060.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.50801.log.gz 2024-05-09T05:02:33.953 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49855.log.gz 2024-05-09T05:02:33.953 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45897.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45897.log.gz 2024-05-09T05:02:33.953 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46538.log 2024-05-09T05:02:33.954 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41168.log 2024-05-09T05:02:33.954 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46538.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49206.log.gz 2024-05-09T05:02:33.954 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52839.log 2024-05-09T05:02:33.954 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46538.log.gz 2024-05-09T05:02:33.954 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41168.log: gzip -5 --verbose -- 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41168.log.gz 2024-05-09T05:02:33.954 INFO:teuthology.orchestra.run.smithi060.stderr: /var/log/ceph/ceph-client.admin.41210.log 2024-05-09T05:02:33.954 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52839.log: gzip -5 --verbose -- 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52839.log.gz 2024-05-09T05:02:33.955 INFO:teuthology.orchestra.run.smithi060.stderr: /var/log/ceph/ceph-client.admin.45939.log 2024-05-09T05:02:33.955 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49813.log 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41210.log.gz 2024-05-09T05:02:33.955 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-05-09T05:02:33.955 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45939.log: gzip -5 --verbose -- 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45939.log.gz 2024-05-09T05:02:33.955 INFO:teuthology.orchestra.run.smithi060.stderr: /var/log/ceph/ceph-mon.a.log 2024-05-09T05:02:33.955 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46889.log 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49813.log.gz 2024-05-09T05:02:33.955 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-05-09T05:02:33.955 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50419.log 2024-05-09T05:02:33.956 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52881.log 2024-05-09T05:02:33.956 INFO:teuthology.orchestra.run.smithi060.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46889.log.gz 2024-05-09T05:02:33.956 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50419.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50419.log.gz 2024-05-09T05:02:33.956 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52313.log 2024-05-09T05:02:33.956 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50633.log 2024-05-09T05:02:33.956 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52881.log.gz 2024-05-09T05:02:33.956 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52313.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52313.log.gz 2024-05-09T05:02:33.956 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50591.log 2024-05-09T05:02:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48540.log 2024-05-09T05:02:33.957 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50633.log.gz 2024-05-09T05:02:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50326.log 2024-05-09T05:02:33.957 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50591.log.gz 2024-05-09T05:02:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48540.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.48540.log.gz 2024-05-09T05:02:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47505.log 2024-05-09T05:02:33.957 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41336.log 2024-05-09T05:02:33.957 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50326.log.gz 2024-05-09T05:02:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47505.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47505.log.gz 2024-05-09T05:02:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46269.log 2024-05-09T05:02:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44778.log 2024-05-09T05:02:33.958 INFO:teuthology.orchestra.run.smithi060.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41336.log.gz 2024-05-09T05:02:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46269.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46269.log.gz 2024-05-09T05:02:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50282.log 2024-05-09T05:02:33.958 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45854.log 2024-05-09T05:02:33.958 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44778.log.gz 2024-05-09T05:02:33.959 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50282.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50282.log.gz 2024-05-09T05:02:33.959 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51178.log 2024-05-09T05:02:33.959 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42896.log 2024-05-09T05:02:33.959 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45854.log.gz 2024-05-09T05:02:33.959 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51178.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51178.log.gz 2024-05-09T05:02:33.959 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51262.log 2024-05-09T05:02:33.959 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42588.log 2024-05-09T05:02:33.959 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42896.log.gz 2024-05-09T05:02:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51262.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51262.log.gz 2024-05-09T05:02:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42890.log 2024-05-09T05:02:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45114.log 2024-05-09T05:02:33.960 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42588.log.gz 2024-05-09T05:02:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48204.log 2024-05-09T05:02:33.960 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42890.log.gz 2024-05-09T05:02:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45114.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45114.log.gz 2024-05-09T05:02:33.960 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42156.log 2024-05-09T05:02:33.961 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53259.log 2024-05-09T05:02:33.961 INFO:teuthology.orchestra.run.smithi060.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48204.log.gz 2024-05-09T05:02:33.961 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50153.log 2024-05-09T05:02:33.961 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53259.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42156.log.gz 2024-05-09T05:02:33.961 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53259.log.gz 2024-05-09T05:02:33.961 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43229.log 2024-05-09T05:02:33.961 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52145.log 2024-05-09T05:02:33.961 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50153.log.gz 2024-05-09T05:02:33.962 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51355.log 2024-05-09T05:02:33.962 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52145.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43229.log.gz 2024-05-09T05:02:33.962 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52145.log.gz 2024-05-09T05:02:33.962 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52042.log 2024-05-09T05:02:33.962 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50867.log 2024-05-09T05:02:33.962 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52042.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.51355.log.gz 2024-05-09T05:02:33.962 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52042.log.gz 2024-05-09T05:02:33.962 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42504.log 2024-05-09T05:02:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46973.log 2024-05-09T05:02:33.963 INFO:teuthology.orchestra.run.smithi060.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50867.log.gz 2024-05-09T05:02:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50675.log 2024-05-09T05:02:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46973.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42504.log.gz 2024-05-09T05:02:33.963 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46973.log.gz 2024-05-09T05:02:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41420.log 2024-05-09T05:02:33.963 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43680.log 2024-05-09T05:02:33.963 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50675.log.gz 2024-05-09T05:02:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50377.log 2024-05-09T05:02:33.964 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41420.log.gz 2024-05-09T05:02:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43680.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43680.log.gz 2024-05-09T05:02:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45709.log 2024-05-09T05:02:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47057.log 2024-05-09T05:02:33.964 INFO:teuthology.orchestra.run.smithi060.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50377.log.gz 2024-05-09T05:02:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50461.log 2024-05-09T05:02:33.964 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47057.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45709.log.gz 2024-05-09T05:02:33.964 INFO:teuthology.orchestra.run.smithi060.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.47057.log.gz 2024-05-09T05:02:33.965 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42294.log 2024-05-09T05:02:33.965 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47311.log 2024-05-09T05:02:33.965 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50461.log.gz 2024-05-09T05:02:33.965 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43215.log 2024-05-09T05:02:33.965 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.0.47311.log: 89.5% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42294.log.gz 2024-05-09T05:02:33.965 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.0.47311.log.gz 2024-05-09T05:02:33.965 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41078.log 2024-05-09T05:02:33.965 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53539.log 2024-05-09T05:02:33.966 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43215.log.gz 2024-05-09T05:02:33.966 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51916.log 2024-05-09T05:02:33.966 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53539.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41078.log.gz 2024-05-09T05:02:33.966 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53539.log.gz 2024-05-09T05:02:33.966 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53301.log 2024-05-09T05:02:33.966 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43228.log 2024-05-09T05:02:33.966 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51916.log.gz 2024-05-09T05:02:33.966 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48831.log 2024-05-09T05:02:33.967 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43228.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53301.log.gz 2024-05-09T05:02:33.967 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43228.log.gz 2024-05-09T05:02:33.967 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48332.log 2024-05-09T05:02:33.967 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49248.log 2024-05-09T05:02:33.967 INFO:teuthology.orchestra.run.smithi060.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48831.log.gz 2024-05-09T05:02:33.967 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.44644.log 2024-05-09T05:02:33.967 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48332.log.gz 2024-05-09T05:02:33.967 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49248.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49248.log.gz 2024-05-09T05:02:33.968 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52755.log 2024-05-09T05:02:33.968 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.2.44644.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50072.log 2024-05-09T05:02:33.968 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.2.44644.log.gz 2024-05-09T05:02:33.968 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46398.log 2024-05-09T05:02:33.968 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52755.log: /var/log/ceph/ceph-client.admin.50072.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52755.log.gz 2024-05-09T05:02:33.968 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47123.log 2024-05-09T05:02:33.968 INFO:teuthology.orchestra.run.smithi060.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50072.log.gz 2024-05-09T05:02:33.968 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46398.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46398.log.gz 2024-05-09T05:02:33.968 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45751.log 2024-05-09T05:02:33.969 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35711.log 2024-05-09T05:02:33.969 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.47123.log: /var/log/ceph/ceph-client.admin.45751.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47123.log.gz 2024-05-09T05:02:33.969 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51874.log 2024-05-09T05:02:33.969 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45751.log.gz 2024-05-09T05:02:33.969 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53217.log 2024-05-09T05:02:33.969 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51874.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35711.log.gz 2024-05-09T05:02:33.969 INFO:teuthology.orchestra.run.smithi060.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51874.log.gz 2024-05-09T05:02:33.969 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46675.log 2024-05-09T05:02:33.970 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47887.log 2024-05-09T05:02:33.970 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53217.log: /var/log/ceph/ceph-client.admin.46675.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53217.log.gz 2024-05-09T05:02:33.970 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46675.log.gz 2024-05-09T05:02:33.970 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45660.log 2024-05-09T05:02:33.970 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42672.log 2024-05-09T05:02:33.970 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45660.log: /var/log/ceph/ceph-client.admin.47887.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47887.log.gz 2024-05-09T05:02:33.970 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45660.log.gz 2024-05-09T05:02:33.970 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46582.log 2024-05-09T05:02:33.971 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43638.log 2024-05-09T05:02:33.971 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42672.log: /var/log/ceph/ceph-client.admin.46582.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42672.log.gz 2024-05-09T05:02:33.971 INFO:teuthology.orchestra.run.smithi060.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46582.log.gz 2024-05-09T05:02:33.971 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51421.log 2024-05-09T05:02:33.971 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53091.log 2024-05-09T05:02:33.971 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43638.log: /var/log/ceph/ceph-client.admin.51421.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43638.log.gz 2024-05-09T05:02:33.971 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52000.log 2024-05-09T05:02:33.971 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51421.log.gz 2024-05-09T05:02:33.972 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.53091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51465.log 2024-05-09T05:02:33.972 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52000.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53091.log.gz 2024-05-09T05:02:33.972 INFO:teuthology.orchestra.run.smithi060.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52000.log.gz 2024-05-09T05:02:33.972 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48391.log 2024-05-09T05:02:33.972 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50717.log 2024-05-09T05:02:33.972 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.48391.log: /var/log/ceph/ceph-client.admin.51465.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51465.log.gz 2024-05-09T05:02:33.972 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48391.log.gz 2024-05-09T05:02:33.972 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44988.log 2024-05-09T05:02:33.973 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.50717.log: -- /var/log/ceph/ceph-client.admin.41252.log 2024-05-09T05:02:33.973 INFO:teuthology.orchestra.run.smithi060.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50717.log.gz 2024-05-09T05:02:33.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41027.log 2024-05-09T05:02:33.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41252.log: 88.8% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41252.log.gz 2024-05-09T05:02:33.973 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44988.log.gz 2024-05-09T05:02:33.973 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52102.log 2024-05-09T05:02:33.973 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46717.log 2024-05-09T05:02:33.973 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41027.log.gz 2024-05-09T05:02:33.974 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52671.log 2024-05-09T05:02:33.974 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46717.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52102.log.gz 2024-05-09T05:02:33.974 INFO:teuthology.orchestra.run.smithi060.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46717.log.gz 2024-05-09T05:02:33.974 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42630.log 2024-05-09T05:02:33.974 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-05-09T05:02:33.974 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52671.log: /var/log/ceph/ceph-client.admin.42630.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52671.log.gz 2024-05-09T05:02:33.974 INFO:teuthology.orchestra.run.smithi060.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42630.log.gz 2024-05-09T05:02:33.974 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44862.log 2024-05-09T05:02:33.975 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52545.log 2024-05-09T05:02:33.975 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44862.log: /var/log/ceph/ceph-mds.a.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44862.log.gz 2024-05-09T05:02:33.975 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41378.log 2024-05-09T05:02:33.975 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.52545.log: 92.2% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-05-09T05:02:33.975 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52545.log.gz 2024-05-09T05:02:33.975 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50507.log 2024-05-09T05:02:33.975 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50195.log 2024-05-09T05:02:33.975 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50507.log: /var/log/ceph/ceph-client.admin.41378.log: 88.6% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41378.log.gz 2024-05-09T05:02:33.976 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50507.log.gz 2024-05-09T05:02:33.976 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49164.log 2024-05-09T05:02:33.976 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53049.log 2024-05-09T05:02:33.976 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.50195.log: /var/log/ceph/ceph-client.admin.49164.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50195.log.gz 2024-05-09T05:02:33.976 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49164.log.gzgzip 2024-05-09T05:02:33.976 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41462.log 2024-05-09T05:02:33.976 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40897.log 2024-05-09T05:02:33.976 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41462.log: /var/log/ceph/ceph-client.admin.53049.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53049.log.gz 2024-05-09T05:02:33.977 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51134.log 2024-05-09T05:02:33.977 INFO:teuthology.orchestra.run.smithi060.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.41462.log.gz 2024-05-09T05:02:33.977 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42420.log 2024-05-09T05:02:33.977 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.51134.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40897.log.gz 2024-05-09T05:02:33.977 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51134.log.gz 2024-05-09T05:02:33.977 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47015.log 2024-05-09T05:02:33.977 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49687.log 2024-05-09T05:02:33.977 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42420.log: /var/log/ceph/ceph-client.admin.47015.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42420.log.gz 2024-05-09T05:02:33.978 INFO:teuthology.orchestra.run.smithi060.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47015.log.gz 2024-05-09T05:02:33.978 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49472.log 2024-05-09T05:02:33.978 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53007.log 2024-05-09T05:02:33.978 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.49687.log: /var/log/ceph/ceph-client.admin.49472.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49687.log.gz 2024-05-09T05:02:33.978 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49472.log.gz 2024-05-09T05:02:33.978 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40733.log 2024-05-09T05:02:33.978 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44946.log 2024-05-09T05:02:33.978 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40733.log: /var/log/ceph/ceph-client.admin.53007.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53007.log.gz 2024-05-09T05:02:33.979 INFO:teuthology.orchestra.run.smithi060.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40733.log.gz 2024-05-09T05:02:33.979 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48089.log 2024-05-09T05:02:33.979 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46183.log 2024-05-09T05:02:33.979 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44946.log: /var/log/ceph/ceph-client.admin.48089.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44946.log.gz 2024-05-09T05:02:33.979 INFO:teuthology.orchestra.run.smithi060.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48089.log.gz 2024-05-09T05:02:33.979 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.46183.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46183.log.gz 2024-05-09T05:02:33.980 INFO:teuthology.orchestra.run.smithi060.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-05-09T05:02:34.056 INFO:teuthology.orchestra.run.smithi060.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-05-09T05:02:34.536 INFO:teuthology.orchestra.run.smithi114.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-05-09T05:02:34.574 INFO:teuthology.orchestra.run.smithi114.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-05-09T05:02:34.576 INFO:teuthology.orchestra.run.smithi114.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-05-09T05:02:34.577 INFO:teuthology.orchestra.run.smithi114.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-05-09T05:02:34.579 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-05-09T05:02:34.579 INFO:teuthology.orchestra.run.smithi114.stderr:real 0m0.787s 2024-05-09T05:02:34.580 INFO:teuthology.orchestra.run.smithi114.stderr:user 0m3.197s 2024-05-09T05:02:34.580 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m0.141s 2024-05-09T05:02:34.591 INFO:teuthology.orchestra.run.smithi060.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-05-09T05:02:34.646 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-05-09T05:02:34.649 INFO:teuthology.orchestra.run.smithi060.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-05-09T05:02:34.658 INFO:teuthology.orchestra.run.smithi060.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-05-09T05:02:34.660 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-05-09T05:02:34.660 INFO:teuthology.orchestra.run.smithi060.stderr:real 0m0.889s 2024-05-09T05:02:34.661 INFO:teuthology.orchestra.run.smithi060.stderr:user 0m3.690s 2024-05-09T05:02:34.661 INFO:teuthology.orchestra.run.smithi060.stderr:sys 0m0.273s 2024-05-09T05:02:34.661 INFO:tasks.ceph:Archiving logs... 2024-05-09T05:02:34.661 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi/7699557/remote/smithi060/log 2024-05-09T05:02:34.662 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T05:02:35.409 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi/7699557/remote/smithi114/log 2024-05-09T05:02:35.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T05:02:35.915 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-09T05:02:35.926 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-09T05:02:35.926 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T05:02:35.929 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T05:02:36.012 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-09T05:02:36.022 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-09T05:02:36.022 DEBUG:teuthology.orchestra.run.smithi060:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T05:02:36.025 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T05:02:36.039 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-05-09T05:02:36.040 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-05-09T05:02:36.043 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T05:02:36.043 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-05-09T05:02:36.043 INFO:teuthology.orchestra.run.smithi060.stdout:^* hv01.front.sepia.ceph.com 3 6 377 52 +2651us[+2642us] +/- 41ms 2024-05-09T05:02:36.044 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 53 -2464us[-2473us] +/- 79ms 2024-05-09T05:02:36.044 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 54 -2222us[-2231us] +/- 67ms 2024-05-09T05:02:36.044 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T05:02:36.046 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T05:02:36.046 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-05-09T05:02:36.046 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv01.front.sepia.ceph.com 3 6 377 52 +2649us[+2652us] +/- 41ms 2024-05-09T05:02:36.046 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 53 -2467us[-2464us] +/- 79ms 2024-05-09T05:02:36.047 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 -2226us[-2226us] +/- 67ms 2024-05-09T05:02:36.047 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T05:02:36.047 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-09T05:02:36.062 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-09T05:02:36.062 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-09T05:02:36.087 DEBUG:teuthology.orchestra.run.smithi060:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T05:02:36.161 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T05:02:36.226 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T05:02:36.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T05:02:36.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T05:02:36.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T05:02:36.285 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-09T05:02:36.295 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-09T05:02:36.361 INFO:teuthology.task.internal:Duration was 558.468870 seconds 2024-05-09T05:02:36.362 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-09T05:02:36.400 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-09T05:02:36.400 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T05:02:36.403 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T05:02:36.442 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T05:02:36.443 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T05:02:36.976 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-09T05:02:36.976 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi060.front.sepia.ceph.com 2024-05-09T05:02:36.976 DEBUG:teuthology.orchestra.run.smithi060:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T05:02:37.000 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2024-05-09T05:02:37.001 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T05:02:37.062 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-09T05:02:37.062 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T05:02:37.065 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T05:02:37.130 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-09T05:02:37.130 DEBUG:teuthology.orchestra.run.smithi060:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T05:02:37.135 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T05:02:37.269 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-09T05:02:37.281 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-09T05:02:37.281 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T05:02:37.307 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T05:02:37.333 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-09T05:02:37.345 DEBUG:teuthology.orchestra.run.smithi060:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T05:02:37.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T05:02:37.371 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = core 2024-05-09T05:02:37.397 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2024-05-09T05:02:37.410 DEBUG:teuthology.orchestra.run.smithi060:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T05:02:37.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T05:02:37.441 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T05:02:37.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T05:02:37.466 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-09T05:02:37.480 INFO:teuthology.task.internal:Transferring archived files... 2024-05-09T05:02:37.480 DEBUG:teuthology.misc:Transferring archived files from smithi060:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi/7699557/remote/smithi060 2024-05-09T05:02:37.481 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T05:02:37.519 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi/7699557/remote/smithi114 2024-05-09T05:02:37.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T05:02:37.555 INFO:teuthology.task.internal:Removing archive directory... 2024-05-09T05:02:37.555 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T05:02:37.558 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T05:02:37.605 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-09T05:02:37.617 INFO:teuthology.task.internal:Not uploading archives. 2024-05-09T05:02:37.617 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-09T05:02:37.643 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-09T05:02:37.644 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T05:02:37.646 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T05:02:37.659 INFO:teuthology.orchestra.run.smithi060.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 05:02 /home/ubuntu/cephtest 2024-05-09T05:02:37.661 INFO:teuthology.orchestra.run.smithi114.stdout: 265552 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 05:02 /home/ubuntu/cephtest 2024-05-09T05:02:37.662 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-09T05:02:37.672 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-09T05:02:37.755 INFO:teuthology.nuke:Checking targets against current locks 2024-05-09T05:02:37.784 DEBUG:teuthology.nuke:shortname: smithi060 2024-05-09T05:02:37.784 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T05:02:37.808 DEBUG:teuthology.nuke:shortname: smithi114 2024-05-09T05:02:37.808 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T05:02:37.812 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi/7699557', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 04:44:35.371617', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDXKlNmw7xjovvOPA5uoX35oogqy+fhjoG7qlpocOBTDo0OuCLVF4iqXfTNQm2BHg0xoBUU7UPRyEg01wz44wcU='} 2024-05-09T05:02:37.837 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-09_04:42:47-fs-wip-63259-3-testing-default-smithi/7699557', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 04:44:35.370115', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIuN6mPI387qZJmyDmYmuGL/2LfVuKpBV7FADGpOtM1VFIMlHQuH6yKHxSB1OYXGM+A1VzzuYysAGwLIo0zDxDA='} 2024-05-09T05:02:37.866 INFO:teuthology.orchestra.console.smithi060:Power off 2024-05-09T05:02:37.866 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-09T05:02:37.896 DEBUG:teuthology.orchestra.console.smithi060:power off output: Chassis Power Control: Down/Off 2024-05-09T05:02:37.896 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T05:02:37.914 INFO:teuthology.orchestra.console.smithi114:Power off 2024-05-09T05:02:37.914 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-09T05:02:37.925 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is on 2024-05-09T05:02:37.941 DEBUG:teuthology.orchestra.console.smithi114:power off output: Chassis Power Control: Down/Off 2024-05-09T05:02:37.941 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T05:02:37.965 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-05-09T05:02:41.926 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T05:02:41.966 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T05:02:42.038 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is on 2024-05-09T05:02:42.077 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-05-09T05:02:46.041 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T05:02:46.078 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T05:02:46.151 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is off 2024-05-09T05:02:46.189 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-05-09T05:02:46.253 INFO:teuthology.orchestra.console.smithi060:Power off completed 2024-05-09T05:02:50.190 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T05:02:50.301 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is off 2024-05-09T05:02:50.401 INFO:teuthology.orchestra.console.smithi114:Power off completed 2024-05-09T05:02:50.507 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/backtrace} duration: 558.4688701629639 failure_reason: 'Test failure: test_backtrace_flush_on_deleted_data_pool (tasks.cephfs.test_backtrace.TestBacktrace)' flavor: default owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=cc1110b8798e4127896dc1850e77f79c status: fail success: false 2024-05-09T05:02:50.508 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-09T05:02:50.595 INFO:teuthology.run:FAIL